Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/PolyU-VCLab/OpenOOD-VLM
Area: Multimodal VLM
Keywords: noisy test-time adaptation, zero-shot classification, out-of-distribution detection, dual distribution estimation, negative-label selection

TL;DR

DDE freezes CLIP, calibrates classification with class inclusion/exclusion Gaussians, and selects negative labels through positive-negative image response differences, improving both classification and rejection in zero-shot noisy test streams and raising average ImageNet harmonic accuracy from AdaND's 73.09% to 76.79%.

Background & Motivation

Vision-language models (VLMs) such as CLIP encode category names as textual prototypes and perform zero-shot classification through image-text similarity. Test-time adaptation uses unlabeled images arriving during deployment to correct mismatches between the pretrained model and the current data. However, many methods assume that every test image belongs to the predefined category set, whereas real streams can contain images outside that set. Applying self-training or cache updates to these out-of-distribution images can turn incorrect predictions into apparently reliable supervision and damage classification on valid inputs. The noisy test-time adaptation (NTTA) task studied here consequently has two online objectives: classify in-distribution (ID) images correctly and reject out-of-distribution (OOD) images. Here, noise primarily means inputs outside the ID semantic label space, rather than simply pixel corruption or mislabeled training data.

The existing zero-shot NTTA method AdaND trains a noise detector during testing, while classification still depends heavily on CLIP's image-text alignment. An image can belong to a category yet lie far from its textual prototype because of pose, background, or appearance differences. Keeping only high-confidence instances does not automatically represent the full visual class distribution, especially the confusing regions between neighboring categories. Detection faces a separate mismatch: negative labels far from ID class names in text space are not necessarily good descriptions of the OOD images in the current stream. Some negative labels are also activated by clean ID images; including many such words in the detection denominator can increase false rejection. Online training additionally requires enough samples and extra optimization, making short streams and low-latency deployment challenging.

The authors therefore estimate visual feature distributions and negative-label response distributions instead of training a more complex discriminator. The former captures how each class appears in the current stream and which neighboring examples should be excluded; the latter identifies labels that distinguish the current ID and OOD images. These statistics serve different decisions: more accurate classification does not automatically imply better recognition of unknown inputs. Core Idea: refine ID decision boundaries using inclusion and exclusion Gaussians, then select OOD negative labels through positive-negative image response differences, jointly improving classification and rejection without updating CLIP parameters.

Method

Overall Architecture

Inputs consist of known ID category names, candidate negative labels, and batches of unlabeled test images; both image and text encoders remain frozen. The system first measures relative support for ID versus negative labels, selects pseudo-positive and pseudo-negative samples, and updates two capacity-limited caches. Positive Feature Distribution Estimation (PFDE) builds classification statistics from reliable ID images, while Negative Label Distribution Estimation (NLDE) uses both positive and negative images to select the detection vocabulary. Adaptive Threshold Decision then converts detection scores into ID/OOD decisions; only accepted ID inputs use the category produced by the fused classifier. Outputs therefore include known-class predictions and OOD scores or rejection decisions for unknown inputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Unlabeled image stream<br/>ID and candidate negative labels"] --> Encode["Frozen CLIP encoding<br/>Initial ID score"]
    Encode --> Cache["Pseudo-positive and pseudo-negative<br/>Dual-cache update"]
    Cache --> PFDE["Positive Feature Distribution Estimation<br/>PFDE"]
    Cache --> NLDE["Negative Label Distribution Estimation<br/>NLDE"]
    PFDE -->|Fused class prediction| Decide["Adaptive Threshold Decision"]
    NLDE -->|ID score after selection| Decide
    Decide --> Output["Accept and classify<br/>or reject as OOD"]

The initial ID score is the total normalized response assigned to all ID labels when ID and negative labels are considered together. A high score indicates stronger support for known categories, not knowledge of the image's true label. The paper selects samples with scores at least 0.75 as pseudo-positive and those below 0.25 as pseudo-negative. Samples between these thresholds do not enter either high-confidence set, but still require predictions as part of the test stream. Each cache has capacity \(Q=1000\); this does not mean storing 1000 images for every class. The following three designs cover classification, negative-label selection, and final decisions; adaptive thresholding is inherited from earlier work and should not be counted as a separate new contribution.

Key Designs

1. Positive Feature Distribution Estimation: model both class membership and likely confusions

PFDE maintains two sets of Gaussian statistics per ID class rather than only one distribution of typical appearances. For class \(k\), pseudo-positive samples whose top-1 CLIP prediction is \(k\) enter the inclusion branch, representing the class's visual variation in the current stream. Pseudo-positive samples whose top-1 prediction is not \(k\), but whose second choice is \(k\), enter that class's exclusion branch and describe nearby confusing regions. The exclusion branch neither treats all OOD images as counterexamples for the class nor simply pools every other ID category. It focuses on ID samples that almost receive the class prediction, providing more specific calibration for local ambiguity around textual prototypes. The same image can provide inclusion evidence for its first-choice class and exclusion evidence for its second-choice class; these are complementary assignments.

The authors estimate class-conditional means and covariances within Gaussian Discriminant Analysis (GDA), computing posteriors with a uniform class prior. The update is described as EM-like: obtain class probabilities, then update statistics using probability-weighted current features and historical effective sample counts. Historical accumulation prevents class estimates from depending only on the current small batch, while new features align distributions with deployment data. To avoid singular covariance inversion when samples are scarce, the method shrinks the covariance toward the identity matrix with coefficient \(10^{-4}\). Some subscripts, minus signs, and denominators are damaged in the extracted formulas, so this note does not reconstruct the complete mean/covariance recurrence or GDA logit expression. The confirmed classification mechanism is that inclusion supplies positive evidence, exclusion subtracts weighted evidence, and the result is fused with CLIP's zero-shot class scores.

The textual definitions on page 8 give the distributional classification score and its sample-dependent fusion weight as:

\[ f_k(x)=f_k^{\mathrm{in}}(x)-\beta f_k^{\mathrm{ex}}(x), \qquad \alpha_t=\min\bigl(\rho B t,\alpha_{\max}\bigr). \]

Here, \(B\) is batch size, \(t\) is the current batch index, \(\rho=0.005\), \(\beta=0.5\), and the cap \(\alpha_{\max}\) is typically 1.0. The system adds \(\alpha_t f_k(x)\) to CLIP's class logit and normalizes the result into fused class probabilities. When data are scarce, distribution estimates are unreliable, so the model relies mainly on the frozen zero-shot prior. Historical visual evidence receives greater influence only as more data accumulate, rather than allowing a few early pseudo-labels to dominate classification immediately. Importantly, this increasing weight depends on the number of processed samples, not on distribution accuracy verified with true labels.

2. Negative Label Distribution Estimation: retain words strong on OOD and weak on ID

Initial candidate negative labels follow the NegLabel approach, selecting labels far from ID class names in a large vocabulary. This text-space filtering is only a starting point and does not guarantee coverage of unknown images in the test stream. NLDE therefore computes each negative label's average normalized image-text response on both the pseudo-negative and pseudo-positive image caches. Normalization considers ID and negative labels together, rather than comparing raw cosine similarities only within the negative vocabulary. A word active on both image groups may describe a shared background or generic visual attribute rather than reliable OOD evidence. Useful labels should respond substantially more strongly to pseudo-negative images and more weakly to pseudo-positive ones.

Page 8 defines this discrimination criterion through a positive-negative response difference, whose meaning is:

\[ \Delta\mathrm{Sim}(y_i^-) =\mathrm{Sim}(\mathcal{X}_{\mathrm{neg}},y_i^-) -\mathrm{Sim}(\mathcal{X}_{\mathrm{pos}},y_i^-). \]

Here, \(\mathrm{Sim}\) is a label's average normalized response over the corresponding image set, not an additional trained predictor. The system retains the top \(\hat M\) negative labels by response difference and recomputes the total ID response using this reduced vocabulary. The default is \(\hat M=500\) for ImageNet and \(\hat M=100\) for fine-grained datasets. NLDE therefore estimates how labels are supported by different image groups; it does not fit another Gaussian to every negative label. PFDE already contains inclusion/exclusion Gaussians, while dual distribution estimation also emphasizes the complementary PFDE and NLDE levels; these distinctions should not be conflated. The subtraction above follows the explicit prose; the exact temperature placement and some symbols in the original normalization expression are incompletely extracted and are not reconstructed here.

3. Adaptive Threshold Decision: turn ranked scores into online rejection

After negative-label selection, the system still needs to decide how much ID support is sufficient to accept an input. A fixed threshold may suit one label space but not another set of classes or OOD sources. The authors adopt the adaptive thresholding idea from OWTTT, using the bimodal score distribution to seek a split with low within-group variance on both sides. Samples scoring at least the threshold are accepted as ID and assigned the fused classifier's prediction; the remaining samples are marked as noise. The 0.75/0.25 cache-selection thresholds serve a different role from this final rejection threshold and should not be treated as the same hyperparameters. Cache selection favors statistical reliability even if it accepts fewer examples, whereas the final decision must cover the entire input stream. Replacing the NLDE score with PFDE's maximum class probability would again conflate which known class is most similar with whether the input belongs to any known class.

A Worked Example

The following hypothetical example explains the mechanism and is not an additional experiment reported by the paper. Suppose the known categories contain two car types, while the current stream also includes boat photographs. A car image with ID score 0.90 exceeds the 0.75 threshold and enters the pseudo-positive cache; if its first choice is class A and second choice is B, it updates A's inclusion branch and B's exclusion branch. A boat image scoring 0.10 is below the 0.25 threshold and enters the pseudo-negative cache, primarily providing evidence for negative-label selection. If “boat” has a much higher average response on pseudo-negative than pseudo-positive images, it is more likely to be retained; if “road” also activates strongly on car images, the response difference suppresses it. Later ambiguous car images can benefit from class distributions closer to the visual data while avoiding rejection caused by irrelevant negative labels. This illustrates cooperation between the two paths, without guaranteeing that the model correctly handles these hypothetical inputs.

Loss & Training

Training-free means that CLIP, prompts, and a new noise detector are not optimized through gradient updates; it does not mean that all model state remains fixed. Online state includes the two caches, class Gaussian statistics, selected negative labels, and the adaptive threshold. The paper's EM-like statistical estimation uses likelihood but introduces no new network-training objective requiring back-propagation. Main experiments use CLIP ViT-B/16; Algorithm 1 selects samples from the current batch, updates statistics, fuses classification, selects negative labels, and computes the rejection threshold. This is consequently an online evaluation that permits adaptation to the current unlabeled batch, not automatically a strict per-sample predict-then-update protocol. The supplied cache does not include supplementary material, leaving initialization details, the exact batch size, cache eviction, and precise metric implementations to be checked against code or the supplement.

Key Experimental Results

Main Results

Under zero-shot and source-free conditions, each ID dataset is paired separately with four OOD sources: iNaturalist, SUN, Texture, and Places. \(\mathrm{Acc}_S\) denotes ID classification accuracy and \(\mathrm{Acc}_N\) denotes noisy detection accuracy; higher is better for both. The paper uses their harmonic mean, \(\mathrm{Acc}_H=2\mathrm{Acc}_S\mathrm{Acc}_N/(\mathrm{Acc}_S+\mathrm{Acc}_N)\), to discourage optimizing classification or rejection alone. The table retains the paper's averages over the four OOD sources rather than recomputing a harmonic mean from average \(\mathrm{Acc}_S\) and average \(\mathrm{Acc}_N\).

Source: Table 2, page 11; CLIP ViT-B/16, average \(\mathrm{Acc}_H\) (%), with gains in percentage points.

ID Dataset	AdaND	DDE	DDE vs. AdaND
ImageNet	73.09	76.79	+3.70
ImageNet-S	53.77	57.83	+4.06
ImageNet-A	56.41	62.98	+6.57
ImageNet-V2	67.72	71.29	+3.57
ImageNet-R	80.25	83.24	+2.99

AdaND is a consistent comparator here, not the strongest alternative on every dataset; for example, DMN and AdaNeg both achieve 55.75% on ImageNet-S. The gain on ImageNet-A supports DDE's usefulness under substantial visual distribution shift, but does not establish applicability to every unknown domain. As a separate detection evaluation, Table 4 reports AUROC and FPR95; the latter is the false-positive rate at a 95% true-positive operating point, for which lower is better.

Source: Table 4, page 13; ImageNet as ID, detection metrics averaged over the four OOD sources (%).

Method	AUROC	FPR95
NegLabel	94.21	25.40
AdaNeg	96.66	18.92
AdaND	95.58	16.00
DDE	97.89	9.80

Compared with AdaND, DDE lowers average FPR95 by 6.20 percentage points; this is not a claim that the number of errors decreases by a relative 6.20%. AUROC measures overall score-ranking discrimination, whereas NTTA harmonic accuracy also depends on the operating threshold and known-class predictions; these metrics are not interchangeable.

Ablation Study

Source: Table 5, page 13; component ablations on clean/noisy ImageNet streams, with all values expressed as accuracy (%). The caption does not explicitly name the noise source; the full model's 65.57/96.25/78.00 matches ImageNet+SUN in Table 2, but this correspondence is not treated as confirmation of the complete protocol.

Config	Clean AccS	Noisy AccS	Noisy AccN	Noisy AccH
Baseline, no added components	66.62	53.43	83.96	65.30
PFDE inclusion only	69.62	61.87	88.34	72.77
PFDE inclusion+exclusion	70.34	62.69	88.90	73.52
NLDE only	66.83	55.72	95.99	70.51
PFDE inclusion+exclusion, NLDE	70.81	64.14	95.76	76.82
Full model, adding adaptive threshold	71.14	65.57	96.25	78.00

Key Findings

• Adding exclusion raises noisy-stream AccH from 72.77% to 73.52%, a 0.75-point gain supporting the value of modeling near-boundary samples.
• NLDE alone raises AccN from 83.96% to 95.99%; the 96.25% result belongs to the full system and cannot be attributed entirely to NLDE.
• With PFDE and NLDE combined, adding the adaptive threshold raises AccH from 76.82% to 78.00%, a further 1.18-point gain.
• Figure 3h and the page 13 discussion show that increasing the negative-label count from the best setting of 500 to 1000 lowers AccH from 78.00% to 71.28%, so more negative labels are not necessarily better.
• Table 7 on page 14 reports 1.84 minutes of total testing time and 5.41 GiB of memory for DDE on an RTX 3090, versus 6.59 minutes and 4.57 GiB for AdaND. DDE is faster here, but does not use less memory.

Source consistency note: page 10 reports a 3.01% gain over AdaND on ImageNet-R, but Table 2's 83.24 and 80.25 differ by 2.99 percentage points. The same page gives ImageNet+iNaturalist AccS/AccN as 67.73/99.42, whereas Table 2 gives 67.72/99.45; numerical comparisons in this note follow the tables without silently harmonizing these discrepancies.

Highlights & Insights

• Turn confusing regions into explicit counterevidence. The exclusion distribution models second-choice confusions around a class rather than a generic average of other classes, supplying boundary information that a single positive prototype cannot express well.
• Let the current image stream determine negative-label value. Semantic distance from ID names alone is insufficient; positive-negative cache response differences provide a training-free discrimination criterion and explain why vocabulary expansion can backfire.
• Use the zero-shot prior as cold-start protection. A sample-dependent fusion weight gradually introduces visual distributions instead of immediately trusting small-sample statistics, an idea transferable to other online prototype methods.

Limitations & Future Work

• Pseudo-labels remain a weak point. This is a reader inference: high confidence does not ensure correctness, and early OOD contamination in the positive cache could damage both classification statistics and negative-label selection.
• Gaussian distributions and uniform priors have limits. Multimodal class distributions, extreme class imbalance, or persistent domain switches may violate the assumptions; the paper does not establish universal robustness under these conditions.
• Negative vocabulary size requires selection. The substantial degradation in Figure 3h makes this more than an incidental implementation detail; adapting vocabulary size to the current stream is worth investigating.
• Efficiency does not imply zero cost. Statistical updates, covariance operations, and caches still consume memory, and Table 7 shows higher memory use than AdaND; substantially larger label spaces require dedicated scaling tests.
• Reproduction details remain incomplete. The supplied full-text cache ends with the references and contains no supplement; damaged formulas, precise metric conventions, and numerical discrepancies limit equation-level reproduction and have not been treated as verified implementations.

Related Work & Insights

• AdaND, source reference [3]: Also targets zero-shot NTTA, but trains a noise detector online. DDE replaces that learning with statistical estimation and additionally uses visual distributions to improve known-class classification.
• NegLabel [18] and AdaNeg [58]: The former provides negative-label detection foundations and the latter uses adaptive negative proxies; DDE further selects discriminative labels using positive-negative image response differences.
• DOTA [13] and DMN [60]: Represent distributional and memory-based VLM adaptation, respectively. DDE combines inclusion/exclusion distributions with OOD-aware negative-label selection rather than only enlarging positive memory.
• OWTTT [24]: Supplies the adaptive thresholding idea for final rejection. This inherited mechanism should be distinguished from the PFDE and NLDE contributions.
• Research direction: Distribution uncertainty could control fusion weights instead of processed sample counts, alongside monitoring cache contamination; this is a follow-up proposal motivated by the method, not an established experimental finding.

Rating

• Novelty: 4/5. Combining inclusion/exclusion distributions with negative-label response differences is targeted, while GDA and adaptive thresholding are established tools.
• Experimental Thoroughness: 4/5. Multiple ID/OOD datasets, backbones, and component analyses are covered, but some protocol details depend on unavailable supplementary material.
• Writing Quality: 3/5. The overall mechanism is understandable, but prose/table discrepancies and the current formula extraction prevent precise reproduction from this cache alone.
• Value: 4/5. Offers a practical statistical approach to online classification and unknown-input rejection with frozen VLMs, subject to deployment checks for cache reliability and resource costs.