VLOD-TTA: Test-Time Adaptation of Vision-Language Object Detectors¶

Conference: ECCV 2026
arXiv: 2510.00458
Code: https://github.com/imatif17/VLOD-TTA
Area: Object Detection
Keywords: Test-Time Adaptation, Vision-Language Object Detection, Entropy Minimization, Prompt Selection, Distribution Shifts

TL;DR¶

To address the performance degradation of vision-language object detectors (such as YOLO-World and Grounding DINO) under test-time distribution shifts, VLOD-TTA utilizes "IoU-weighted entropy minimization" to focus adaptation on spatially dense clusters of overlapping bounding box proposals, and "image-conditioned prompt selection" to select the text prompts that best match the current image. By updating only a minimal number of adapter parameters, it consistently outperforms existing TTA baselines and previous specialized methods under diverse shifts—including artistic styles, adverse driving conditions, low-light, and common degradations—at a much lower computational cost.

Background & Motivation¶

Vision-Language Object Detectors (VLODs) have recently achieved strong zero-shot generalization by aligning regional features and text through large-scale image-text pre-training, enabling the detection of novel categories given arbitrary category text. YOLO-World even achieves real-time performance. However, once moved out of the pre-training distribution—such as transitioning from natural photos to watercolors, day to night, or encountering motion blur and compression noise—their localization and classification accuracy degrade significantly. Offline source-free domain adaptation can alleviate this, but it requires pre-collecting target domain data and performing offline training, which is often impractical in real-world deployments. Since the environment changes online, the model must adapt on the fly during inference. This points to Test-Time Adaptation (TTA), which adapts the model online using only unlabeled test data.

The limitation is that while TTA has been extensively studied for vision-language classification, it remains almost unexplored for VLODs. The only precedent, TTAOD-F, employs a framework of "multi-modal prompts + mean teacher + memory-augmented pseudo-labels." Although effective, its overhead is heavy: it requires maintaining an extra teacher detector, running an additional forward pass for pseudo-label generation, and using DINOv2 features for memory retrieval, which drives up test-time GPU memory and latency. Furthermore, it is tailored for transformer-based detectors and cannot be easily ported to CNN detectors like YOLO-World. Another lighter alternative is to directly adopt entropy minimization from classification. However, directly applying this to detection suffers from two critical flaws: first, it only sharpens the highest category scores, easily making poorly localized bounding boxes overconfident and amplifying confirmation bias; second, it treats all bounding box proposals equally, assigning the same weight to isolated proposals, cross-instance proposals, and clusters of spatially consistent, overlapping proposals. As illustrated by the paper's Figure 1, standard entropy minimization boosts the scores for both a person and a dog, ultimately producing a false positive detection of a non-existent dog. Moreover, the common VLM practice of "averaging multiple template prompts" is highly ineffective or even detrimental for VLODs: as shown in Figure 2, prompt averaging suppresses the score of a person below the detection threshold, leading to a missed detection.

Consequently, the authors identify an overlooked structural signal: modern detectors output hundreds of densely overlapping bounding box proposals per image, which are essentially redundant partial views of the same object. True objects are typically surrounded by a dense cluster of proposals with consistent predicted categories and high mutual overlap, whereas false positives tend to be scattered and isolated. Core Idea: Re-weight entropy using the IoU overlap structure among proposals to bias adaptation toward spatially consistent proposal clusters (IWE), while selecting the most relevant text prompts for each category based on image content rather than blindly averaging them (IPS). Combining these two and tuning only a lightweight adapter significantly enhances the robustness of VLODs under distribution shifts with exceptionally low overhead.

Method¶

Overall Architecture¶

The goal of VLOD-TTA is to adapt the detector online using only the information from a single test image and a set of category prompts from an unknown target domain, and immediately produce the detection results using the adapted model. The framework consists of only two trainable components stacked on top of a frozen VLOD: Image-Conditioned Prompt Selection (IPS) selects the most useful text prompts for the current image during the scoring stage and fuses them with the detector's original scores; IoU-Weighted Entropy Minimization (IWE) structures the adaptation target by weighting these fused proposal scores based on their spatial overlap.

Specifically, during a single inference step, the input image is first processed by the visual detector to obtain \(N\) proposal bounding boxes and their region features, while the category prompt pool is processed by the text encoder to obtain embeddings. For each category, IPS computes the compatibility of each prompt with the current image, retaining only the top-\(\rho\) prompts. Their similarities are averaged to obtain the category score for each proposal, which is then fused with the detector's base scores via weighted fusion. Next, the top-\(M\) highly confident proposals are selected based on the fused scores to construct an intra-class IoU graph, yielding a weight for each box. IWE uses these weights to compute a weighted sum of proposal entropies as the loss, performing only a single backward step to update the adapter and a text residual vector. Because the adaptation is performed on a single image, the updated parameters are not expected to generalize to other images; thus, the adapter is reset to zero initialization after each prediction before processing the next image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Test Image + Category Prompt Pool"] --> B["Visual Detector<br/>N Proposals + Region Features"]
    A2["Text Encoder<br/>Prompt Embeddings"] --> C
    A --> A2
    B --> C["Image-Conditioned Prompt Selection (IPS)<br/>Select Top-ρ Prompts & Fuse with Base Scores"]
    C --> D["Get Top-M Proposals<br/>Build Intra-class IoU Graph"]
    D --> E["IoU-Weighted Entropy Minimization (IWE)<br/>Weigh Entropy by Cluster Size"]
    E -->|One-step Backprop<br/>Update Adapter + Text Residual| F["Adapted Model Outputs Predictions<br/>Then Reset Adapter"]

Key Designs¶

1. IoU-Weighted Entropy Minimization (IWE): Using Overlapping Cluster Size as Weights to Focus Entropy Minimization on Confident Regions

Standard entropy minimization treats each proposal independently, assigning equal weights to both isolated proposals and clusters of spatially consistent, overlapping proposals. Consequently, it aggressively sharpens predictions even in unreliable regions, amplifying confirmation bias—the root cause of producing false positives (like the dog). IWE addresses this by first determining the predicted category of each proposal \(\hat{c}_i=\arg\max_k p_{i,k}\). An IoU graph \(G_c\) is constructed separately for each category \(c\), where vertices are proposals predicted as \(c\), and an edge is connected between two proposals if their \(\mathrm{IoU}\ge\theta\). This ensures that the dense cluster of mutually overlapping proposals around actual objects forms a large connected component, while isolated proposals form small components. Let \(\mathcal{C}(i)\) denote the connected component containing proposal \(i\), and \(|\mathcal{C}(i)|\) represent its size. The weight is defined as \(w_i=|\mathcal{C}(i)|^{\gamma}\), where the exponent \(\gamma\ge0\) controls the influence of the cluster size. The final objective is the weighted average entropy:

\[\mathcal{L}_{\mathrm{IoU\text{-}Ent}}=\frac{\sum_{i=1}^{N} w_i\,\mathcal{H}(\mathbf{p}_i)}{\sum_{i=1}^{N} w_i}.\]

A key detail is that the weights depend solely on the IoU graph and are treated as constants during backpropagation; gradients flow only through the entropy \(\mathcal{H}(\mathbf{p}_i)\). Consequently, larger clusters (which are more likely to be true objects) dominate the optimization while isolated proposals are suppressed. This forces the model to become confident only in densely consistent regions, effectively suppressing the "boost-everything" confirmation bias of standard entropy. The authors also verify that this IoU-weighting principle can be transferred to pseudo-label-based TTA (IWPL), demonstrating that it is a general concept rather than unique to entropy minimization.

2. Image-Conditioned Prompt Selection (IPS): Selecting the Best-Matching Prompts per Image to Replace Blind Prompt Averaging

VLMs commonly use "averaging multiple templates per category" to boost robustness, but the authors find that this yields limited gains or even harms performance on VLODs. Prompt averaging flattens useful signals along with noise; for example, in Figure 2, it suppresses the score of a person below the threshold, leading to a missed detection. IPS replaces this with an image-conditioned selection: for each prompt \(t\) of category \(k\), it first computes the average cosine similarity between \(t\) and all proposal features of the current image, \(r_{k,t}=\frac{1}{N}\sum_i \hat{\mathbf{v}}_i^\top \hat{\mathbf{e}}_{k,t}\), as the "image-conditioned compatibility." This measures how well the prompt aligns with the content of the current image (the authors prove in the appendix that maximizing \(r_{k,t}\) is equivalent to minimizing the mean squared Euclidean distance between normalized features). Then, only the top-\(\rho\) compatible prompts \(\mathcal{S}_k\) are retained for each category. The proposal category scores are averaged only over these selected prompts: \(\tilde{z}_{i,k}=\frac{1}{|\mathcal{S}_k|}\sum_{t\in\mathcal{S}_k} z_{i,k,t}\).

To further align text and region embeddings, IPS adds a learnable residual vector \(\Delta\) in the text embedding space, modifying each prompt embedding to \(\tilde{\mathbf{e}}_{k,t}=(\mathbf{e}_{k,t}+\Delta)/\lVert\mathbf{e}_{k,t}+\Delta\rVert_2\), which is one of the few parameters updated at test-time. Finally, the selected scores are fused with the detector's base scores \(s_{i,k}\) via weighted fusion: \(g_{i,k}=\lambda\tilde{z}_{i,k}+(1-\lambda)s_{i,k}\). Retaining the base scores instead of solely using the selected prompts is crucial because the early vision-text fusion in VLODs makes regional features implicitly dependent on the text embeddings; an excessively large \(\lambda\) would discard information carried by the detector's original prompts. In experiments, YOLO-World uses \(\lambda=0.3\), while Grounding DINO, which features earlier and deeper multi-modal fusion, uses only \(\lambda=0.1\).

3. Single-Image Zero-Initialized Adapter Update: Placing Adapters Based on Architectural Differences with One-Step Adaptation followed by Reset

In principle, VLOD-TTA can tune any subset of parameters. However, for lightweight computation and to prevent overfitting, it freezes the backbone and updates only a minimal set of adapter parameters \(\Phi\) and the text residual \(\Delta\), all initialized to zero at the start of the test phase. The placement of these modules is determined by the detector's architecture: YOLO-World's visual detector and text encoder are largely decoupled, interacting only during the final scoring stage, so Conv-Adapters are inserted into the visual backbone and neck (ablations show that tuning the backbone alone yields +4.4 AP50, while tuning the text encoder is almost useless). Conversely, Grounding DINO features early cross-modal fusion, where text features penetrate both localization and classification. Thus, MLP-Adapters are inserted only into the text encoder (tuning it yields +3.3 AP50, whereas tuning the visual encoder leads to performance drops, as its 172M parameters are too large and prone to overfitting on a single image).

In terms of workflow, once the fused scores \(g_{i,k}\) are obtained, the top-\(M\) proposals are selected based on \(\max_k g_{i,k}\) (to accelerate IoU graph construction). A single step of adaptation is then performed using the IWE objective, updating \((\Phi,\Delta)\) via backpropagation. Since adaptation is performed on a single test image, and the updated parameters are not expected to generalize across images, \((\Phi,\Delta)\) are reset to zero initialization after each prediction, restarting from scratch for the next image. This "zero-initialization + single-step + reset-per-image" pipeline ensures that adaptation on each image remains independent, while minimizing test-time overhead to a single forward and backward pass.

Loss & Training¶

The sole adaptation loss is the weighted entropy \(\mathcal{L}_{\mathrm{IoU\text{-}Ent}}\) from IWE (Eq. 3). Key hyperparameters: IoU threshold \(\theta\) (for graph construction, stable within 0.5–0.7), cluster size exponent \(\gamma=1.1\) (stable within 0.6–1.6), top-\(M\) proposals \(M=600\), fusion coefficient \(\lambda\) (0.3 for YW / 0.1 for GD), and prompt selection ratio \(\rho=0.25\). Designed for real-time deployment, adaptation is performed with a batch size of 1 per image for a single step. For each category, ChatGPT is used to generate \(T=16\) dataset-agnostic prompts. Zero initialization is applied to the adapters to ensure that the starting performance is equivalent to the pre-trained functions.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on YOLO-World and Grounding DINO across six domain shift datasets (Watercolor / ClipArt / Comic for artistic styles, Cityscapes / BDD100K for driving, ExDark for low light) and two degradation benchmarks (PASCAL-C / COCO-C, featuring 15 degradations \(\times\) 5 severity levels), totaling 96 test scenarios.

Dataset (YW)	Metric	ZS	Strongest Baseline (Adapter)	VLOD-TTA
Watercolor	AP50	47.9	51.5	53.1
ClipArt	AP50	40.1	44.1	45.4
Comic	AP50	29.4	34.7	36.1
BDD100K	mAP	13.3	13.7	14.6
PASCAL-C	AP50 Mean	34.6	37.0	38.5

Under style shifts, VLOD-TTA yields average improvements over ZS of +3.3 mAP / +5.8 AP50 for YW, and +2.4 mAP / +3.3 AP50 for GD. On PASCAL-C, it achieves optimal performance across all degradations, with the largest gains observed in JPEG compression (+8.2) and glass blur (+7.0). On GD, ClipArt mAP increases from 38.4 to 41.2, and Comic mAP rises from 31.2 to 34.2, displaying larger improvements than YW.

Ablation Study¶

Configuration (YW, Mean of 3 Style Datasets)	AP50	Description
ZS	~39.1	Zero-shot baseline
Adapter (Standard Entropy)	~43.4	Standard entropy minimization
+ IWE	~44.3	Switched to IoU-weighted entropy, consistent gains
Full (IWE + IPS)	~44.9	Combination of both yields the best results

Efficiency comparison (COCO-C / GD) is particularly crucial: VLOD-TTA achieves an average mAP of 26.2, which is slightly higher than the 26.0 of the previous-generation TTAOD-F. However, it is significantly more efficient across the board, with total parameters of 173.9M vs 629.9M, latency of 531.6 vs 701.9 ms/img, and GPU memory of 3.76 vs 11.37 GB. Utilizing the same warm-start initialization as TTAOD-F yields VLOD-TTA, which further increases mAP to 27.3, outperforming TTAOD-F on nearly all degradation categories. Trainable parameters are only 0.89M (0.95M for VLOD-TTA).

Key Findings¶

Both components make positive, complementary contributions: using IWE alone consistently outperforms standard entropy Adapter, and adding IPS yields further gains. While prompt averaging drops AP50 on Watercolor/Comic (-0.35 / -0.30), prompt selection yields improvements on all three datasets (averaging +1.0).
IWE is not uniquely beneficial to entropy minimization: incorporating IoU-weighting into pseudo-label-based TTA (IWPL) also consistently outperforms standard pseudo-labeling, showing that "weighting by overlapping clusters" is a general principle.
Since the target domain is unknown, adjusting learning rates per dataset is impractical. Thus, an adapter is used (a single learning rate of 5e-3 generalizes well across datasets) instead of Batch Normalization (which requires per-dataset tuning of 1e-2/3e-2). However, the VLOD-TTA objective remains effective for BN tuning, showing it is not bound to a specific parameter subset.
Performance is weaker on Cityscapes: "rider" and "person" labels are easily confused (IWE tends to push ambiguous "rider" predictions into "person" clusters), small objects are abundant, and the default 640 resolution downsamples distant vehicles/pedestrians excessively. By merging rider \(\to\) person and increasing the resolution to 1024, the mAP gain of VLOD-TTA on Cityscapes expands from +0.6 to +2.7 mAP, confirming that the lack of stable overlapping proposals for small objects is the primary bottleneck.

Highlights & Insights¶

Transforming "densely overlapping proposals" from a redundant burden into a useful signal: proposals around true objects cluster into large components, while false positives remain scattered. Using the connected component size as entropy weights directly suppresses confirmation bias from the start—a simple yet profound observation that addresses the core challenge of detection-based TTA.
Treating weights as constants and propagating gradients only through entropy is an elegant engineering choice: it exploits spatial structures without allowing non-differentiable graph construction to block backpropagation.
The "image-conditioned compatibility" of IPS provides a quantitative rationale for why prompts should be selected per image rather than averaged, backed by theoretical equivalence between cosine similarity and Euclidean distance. This idea is transferable to prompt engineering in any open-vocabulary task.
Placing adapters based on architectural differences (tuning vision in YW, tuning text in GD) reflects a deep understanding of the nature of early vs. late fusion, avoiding blind adapter insertion.

Limitations & Future Work¶

The authors acknowledge that in scenarios where small objects are dense and overlapping proposals are sparse (e.g., Cityscapes), the benefits of IWE are constrained, requiring mitigation strategies like raising resolution or merging semantically overlapping classes.
It remains slower than zero-shot inference due to the required test-time backpropagation; the authors envision gradient-free adaptation to further reduce latency.
Single-image adaptation combined with per-image resetting means that cross-image knowledge is not accumulated, which may underutilize sequential, identically distributed stream data. Additionally, the prompt pool is generated by a fixed GPT model; while dataset-agnostic prompts are highly practical, they are suboptimal compared to dataset-specific prompts (which are unrealistic to obtain and thus not used).
For classes with highly overlapping semantics (like "rider" and "person"), IWE tends to push ambiguous cases toward the dominant class, potentially exacerbating fine-grained confusion.

vs TTAOD-F: Both are specialized TTA methods for VLODs. However, TTAOD-F employs a mean teacher + dual-prompt tuning + DINOv2 memory framework, requiring the maintenance of a second detector and extra forward passes, which is heavy and restricted to transformer architectures. In contrast, VLOD-TTA relies on a single-step entropy objective with no teacher forward pass, supports both CNN and transformer architectures, and achieves slightly higher accuracy with significantly reduced overhead.
vs Tent / Standard Entropy Minimization: Tent treats proposals independently and only sharpens the highest class scores, easily inducing overconfidence in unreliable regions. VLOD-TTA uses IoU structures to weight adaptation, sharpening only on spatially consistent clusters.
vs CLIP-style Prompt Averaging: CLIP improves zero-shot accuracy through multi-template averaging, but averaging yields marginal benefits or even degrades performance in VLODs. IPS instead selects the most relevant prompts per image and fuses them with base scores.
vs TPT / VPT / DPE: These prompt/cache-based TTA methods adapted from classification operate at the image level without considering regional proposals. VLOD-TTA performs proposal-level adaptation, simultaneously handling localization and classification.

Rating¶

Novelty: ⭐⭐⭐⭐ The first lightweight, unified CNN/transformer TTA method for VLODs. The IoU-weighted entropy approach is highly clever and addresses an unexplored gap.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 datasets + 15 degradations \(\times\) 5 severity levels + dual detectors totaling 96 scenarios, along with thorough ablations on efficiency, backbones, specialized domains, and failure cases.
Writing Quality: ⭐⭐⭐⭐ Clear motivation illustrations, comprehensive methodology explanations, and honest failure case analysis.
Value: ⭐⭐⭐⭐ Real-time deployment-friendly, offering a significantly better accuracy-efficiency trade-off. The IoU-weighting principle is also easily transferable to pseudo-labeling-based TTA.