Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Evelyn1ywliang/PuRF-MLTTA
Area: Multimodal VLM
Keywords: Test-Time Adaptation (TTA), Vision-Language Models, Multi-Label Recognition, Region Purification, Cache Mechanism
TL;DR¶
To overcome shared global representation entanglement and subsequent cache saturation in multi-label test-time adaptation (MLTTA) of vision-language models, PuRF introduces region purification combined with episodic prototype anchoring and temporal decay refreshing, achieving fine-grained multimodal adaptation and persistent streaming adaptability without backbone fine-tuning.
Background & Motivation¶
Test-time adaptation (TTA) using foundation vision-language models like CLIP has emerged as an effective paradigm for mitigating distribution shifts on unlabeled test streams, branching mainly into prompt-tuning strategies and cache-based memory retrieval. However, while real-world visual inputs predominantly feature multiple co-occurring objects, existing TTA advances have focused almost exclusively on single-label categorization. Extending conventional cache-based frameworks to the more realistic multi-label test-time adaptation (MLTTA) setting immediately hits a fundamental one-to-many representation coupling dilemma.
Under typical global pooling, a single visual feature vector entangles all visible object classes. In multi-label scenes, visually salient or large-scale categories (such as water or sky) dominate this shared embedding, severely diluting signals from co-occurring or minor objects. Storing such coupled global vectors into multiple class-wise cache slots contaminates category prototypes, misaligning class decision boundaries during retrieval. Although integrating local region crops can theoretically isolate category-specific evidence, unconstrained test-time random crops carry substantial background noise, partial cuts, and irrelevant context. Furthermore, because a multi-label image generates candidate entries across multiple categories simultaneously, memory buffers saturate rapidly. The entropy admission threshold escalates so quickly that later, better-adapted stream samples are blocked, freezing the cache into early-stage suboptimality.
To reconcile the need for isolated local evidence with the risk of cache contamination and saturation, this work establishes a purification-driven framework named PuRF. Core idea: exploit relative cross-class activations to filter noise-free local regions for fine-grained multi-label supervision, while coupling episodic prototype anchoring with exponential temporal decay to build discriminative, dynamically refreshing test-time memory caches.
Method¶
Overall Architecture¶
For incoming streaming unlabeled images, PuRF first extracts multi-scale stochastic crops alongside global augmented views. It computes class-relative confidence across region candidates to filter out ambiguous patches, generating a purified region pool that intersects with global Top-\(\kappa_g\) predictions to derive robust pseudo-labels. In the memory update stage, for each positive pseudo-label, PuRF anchors the lowest-entropy purified region into the respective class cache slot (episodic purification). To safeguard cache dynamism across long test streams, it penalizes long-retained entries via exponential time decay (temporal refreshing). Finally, predictions are produced via global-local max aggregation and lightweight text residual tuning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Test Image x & Streaming Input"] --> B["Global Augmented Views Top-ฮบg<br/>Global Candidate Set G"]
A --> C["Stochastic Multi-Scale Crops Q"]
C --> D["Region Purification: Relative Activation Selection<br/>Purified Region Feature Set F_V^pur"]
B & D --> E["Global-Local Prediction Intersection<br/>Reliable Pseudo-Label Vector y~"]
D & E --> F["Episodic Purification: Minimum-Entropy Anchoring<br/>Class-Specific Cache Prototypes M^c"]
F --> G["Temporal Refreshing: Exponential Time-Decay Penalty<br/>Penalize Aging Entries & Refresh Cache Slots"]
E & F --> H["Global-Local Max Aggregation with BCE & Alignment<br/>Optimize Textual Residuals & Predict Multi-Labels"]
Key Designs¶
1. Region Purification: Decoupling Local Evidence via Relative Activation and Adaptive Thresholds
Global embeddings are contaminated by salient categories, while raw local crops suffer from severe background redundancy. PuRF passes \(Q\) local candidate features through cross-class Softmax normalization against text embeddings \(\hat{s}_q^c = \text{softmax}_c(\langle \mathbf{f}_{\mathrm{V}}^{(q)}, \mathbf{F}_{\mathrm{T}}^c \rangle)\) to quantify class dominance. A candidate is admitted into the purified region set \(\mathcal{F}_{\mathrm{V}}^{\text{pur}}\) only if its dominant class confidence exceeds that class's running adaptive threshold \(\mu_c\):
This moving average compensates for class imbalance and varying intrinsic confidence across categories. The purified regions yield a local candidate set \(\mathcal{R}\) via top-1 consensus, which intersects with the global augmented candidate set \(\mathcal{G}\) to form the final multi-label supervision \(\tilde{y} = \mathcal{G} \cap \mathcal{R}\).
2. Episodic Purification: Class-Specific Prototype Anchoring and Max-Pooling Retrieval
Storing a single global embedding across all predicted positive classes corrupts category-specific visual prototypes. Instead, for each identified positive pseudo-label \(\tilde{y}_i \in \tilde{y}\), episodic purification identifies the single purified region displaying the lowest prediction entropy:
This anchor feature \(\mathbf{f}_{\mathrm{V}}^{(r^{\star})}\) is inserted into class cache \(\mathcal{M}^{\tilde{y}_i}\) with its entropy value. During inference, cache retrieval replaces global dot-products with maximum-activation pooling across purified regions \(\max_{\mathbf{f}_{\mathrm{V}}^{(q)} \in \mathcal{F}_{\mathrm{V}}^{\text{pur}}} \langle \mathbf{f}_{\mathrm{V}}^{(q)}, \mathbf{F}_{\text{cache}}^c \rangle\), thereby isolating subtle, non-dominant object signals from distracting backgrounds.
3. Temporal Refreshing: Mitigating Cache Saturation with Exponential Retention Decay
Because multi-label images supply multiple cache pairs per step, memory queues (capacity \(L=3\)) saturate rapidly, raising the admission bar and locking out later test samples. Consequently, caches freeze, dominated by early prototypes formed before the model converged. Temporal refreshing tracks the cumulative residence steps \(t_i\) of each cached entry and penalizes retention duration exponentially:
When residence time \(t_i\) surpasses threshold \(\delta\) (default \(\delta=1000\)), the inflation factor \(w_i^{\text{time}} > 1\) amplifies the effective entropy \(\mathcal{H}_i^{\prime}\). This lowers the retention priority of obsolete items, allowing better-adapted, higher-quality prototypes from later test stages to refresh the memory.
Loss & Training¶
PuRF freezes the CLIP image and text encoders, updating only lightweight textual residual vectors \(\Delta \mathbf{F}_{\mathrm{T}}^c\) alongside multi-expert text expansions. The optimization objective combines multi-label binary cross-entropy, marginal prediction entropy, and cross-modal prototype alignment:
Here, \(\mathcal{L}_{\text{BCE}}\) supervises the unified prediction logit \(s_{\text{TTA}}^c = \frac{1}{2}\langle \mathbf{F}_{\mathrm{V}}, \mathbf{F}_{\mathrm{T}}^c \rangle + \frac{1}{2}\max_{\mathbf{f}_{\mathrm{V}}^{(q)}}\langle \mathbf{f}_{\mathrm{V}}^{(q)}, \mathbf{F}_{\mathrm{T}}^c \rangle + s_{\text{cache}}^c\) against pseudo-labels \(\tilde{y}\), with balancing weights set to \(\lambda_1 = 0.2\) and \(\lambda_2 = 0.5\). Running textual representations are progressively accumulated and \(\ell_2\)-normalized across streaming steps.
Key Experimental Results¶
Main Results¶
Evaluated on five standard multi-label benchmarksโPASCAL VOC 2007/2012, MS-COCO 2014/2017, and NUS-WIDEโPuRF significantly outperforms both prompt-based and cache-based state-of-the-art methods across all backbones. The table below highlights performance under ViT-B/16 and ViT-B/32 architectures in terms of mAP (%):
| Backbone & Method | VOC 2007 | VOC 2012 | COCO 2014 | COCO 2017 | NUS-WIDE | Overall Avg. |
|---|---|---|---|---|---|---|
| ViT-B/16: Zero-shot CLIP | 79.58 | 79.25 | 54.42 | 54.13 | 45.65 | 62.61 |
| ML-TTA (ICLR'25, Prompt-based) | 81.28 | 81.13 | 57.52 | 57.49 | 46.55 | 64.80 |
| DPE (NeurIPS'24, Prototype Evolving) | 84.10 | 83.62 | 59.21 | 59.28 | 48.90 | 67.02 |
| ReTA (ACM MM'25, Single-label Cache) | 85.09 | 84.49 | 60.37 | 60.30 | 49.20 | 67.89 |
| PuRF (Ours, ViT-B/16) | 88.18 | 87.12 | 66.05 | 65.33 | 50.72 | 71.48 |
| Gain over ReTA / Gain over ML-TTA | +3.09 | +2.63 | +5.68 | +5.03 | +1.52 | +3.59 / +6.68 |
| ViT-B/32: Zero-shot CLIP | 77.18 | 76.85 | 50.31 | 50.15 | 42.90 | 59.48 |
| ReTA (ACM MM'25, ViT-B/32) | 83.99 | 83.51 | 57.02 | 56.58 | 48.05 | 65.83 |
| PuRF (Ours, ViT-B/32) | 87.32 | 86.04 | 63.42 | 63.02 | 49.61 | 69.88 |
| Gain over ReTA (ViT-B/32) | +3.33 | +2.53 | +6.40 | +6.44 | +1.56 | +4.05 |
Ablation Study¶
Component ablations on ViT-B/16 (Table 5 in the original paper) demonstrate the mutual synergy of Region Purification (RP), Episodic Purification (EP), and Temporal Refreshing (TR):
| Config | RP (Region) | EP (Cache) | TR (Cache) | VOC 2007 (mAP %) | COCO 2017 (mAP %) | Note |
|---|---|---|---|---|---|---|
| 1 (Baseline) | โ | โ | โ | 84.83 | 60.71 | Standard global residual TTA with vanilla cache |
| 2 (+RP) | โ | โ | โ | 86.67 | 63.35 | Purified regional cues uncouple co-occurring objects (+2.64% COCO) |
| 3 (+EP) | โ | โ | โ | 85.79 | 62.59 | Anchored class-specific prototypes clean cache (+1.88% COCO) |
| 4 (+RP+EP) | โ | โ | โ | 87.50 | 64.56 | Complementary fine-grained visual-memory alignment |
| 5 (+RP+TR) | โ | โ | โ | 87.36 | 64.12 | Mitigating stream saturation without episodic cache |
| 6 (+EP+TR) | โ | โ | โ | 86.65 | 63.37 | Dual cache purification without region extraction |
| 7 (Full PuRF) | โ | โ | โ | 88.18 | 65.35 | Full model achieves peak multi-label accuracy (+4.64% COCO) |
Further analysis confirms that relative-confidence selection (65.35%) outperforms raw all-region aggregation (63.46%) and absolute thresholding (65.02%). In cache aggregation, max pooling (65.35%) clearly surpasses average pooling (64.92%).
Key Findings¶
- Disproportionate Gains on Complex Multi-Object Benchmarks: PuRF delivers its largest margins on MS-COCO (+5.03% to +6.44% mAP over ReTA), where dense co-occurrence is frequent, confirming that region purification successfully addresses object entanglement.
- Superior Efficiency and Lower Memory Footprint: Unlike ML-TTA which incurs heavy backpropagation and prompt re-initialization (3.05 FPS, 5.77 GB GPU memory), PuRF achieves nearly double the throughput (5.74 FPS) with only 1.04 GB GPU RAM on a single RTX 3090.
- Cross-Architecture Generalization: Evaluated on next-generation VLMs including EVA-02, SigLIP2, and MetaCLIP2, PuRF consistently outperforms prior cache baselines by 3.56% to 3.69% average mAP, hitting 84.76% average mAP on SigLIP2.
Highlights & Insights¶
- Cross-Class Relative Normalization: Employing cross-class Softmax dominance rather than absolute visual similarity sidesteps the challenge of uneven prediction confidence across imbalanced multi-label classes.
- Time-Decay Cache Scheduling: Adapting learning rate scheduling principles to test-time feature retention elegantly solves cache freezing using a single-line exponential penalty.
- Broad Transferability: The dual principles of relative region purification and time-penalized streaming memory readily generalize to open-vocabulary object detection (OVOD) and streaming video question answering.
Limitations & Future Work¶
- Reliance on Stochastic Cropping: Sampling 50 random rectangular crops can fragment fine-grained or elongated objects. Coupling with lightweight class-agnostic object proposal networks or SAM-based masks could improve region precision.
- Fixed Stream-Length Prior: The temporal decay constant \(\delta\) is manually preset to 1000, assuming knowledge of the test stream scale. Dynamic decay schedules keyed to streaming redundancy metrics warrant future exploration.
Related Work & Insights¶
- vs ML-TTA (ICLR 2025): While ML-TTA optimizes prompt vectors with bound entropy minimization per sample without memory re-use, PuRF establishes a purified cache architecture that preserves historical insights, running twice as fast while beating ML-TTA by 6.68% mAP on ViT-B/16.
- vs ReTA / DPE (ACM MM 2025 / NeurIPS 2024): State-of-the-art single-label cache methods fail in multi-label scenarios due to salient class dominance in shared global vectors; PuRF isolates class evidence into clean prototypes and prevents streaming saturation.
Rating¶
- Novelty: โญโญโญโญ [Well-motivated formulation of multi-label representation coupling and cache saturation with clean purification designs]
- Experimental Thoroughness: โญโญโญโญโญ [Exhaustive evaluations across 5 datasets, 4 CLIP architectures, 3 advanced foundation VLMs, and 3 prompt formats]
- Writing Quality: โญโญโญโญโญ [Clear problem setup, intuitive figures, and sound theoretical and empirical grounding]
- Value: โญโญโญโญโญ [Sets a high-performance, efficient benchmark for multi-label test-time adaptation]