Task-Agnostic Incremental Vision-Language Object Detection via Prompt Augmentation and Distribution-Aware Fusion

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yonghanjiang/TADA
Area: Object Detection
Keywords: Continual Learning, Open-Vocabulary Object Detection, Prompt Interference, Parameter Conflict, Mahalanobis Distance

TL;DR

To tackle cross-task prompt interference and LoRA parameter conflicts in incremental vision-language object detection without oracle task priors, TADA introduces training-time stochastic prompt augmentation and test-time Mahalanobis-distance distribution-aware fusion, achieving 64.5 mAP on the ODinW-13 benchmark (+32.0 mAP over DitHub).

Background & Motivation

Vision-Language Object Detection (VLOD) built upon large-scale multimodal pre-training has achieved impressive open-vocabulary detection capabilities. However, deploying pre-trained models into specialized downstream domains frequently leads to substantial performance degradation caused by domain distribution shifts. Incremental Vision-Language Object Detection (IVLOD) addresses this challenge by continuously learning novel concepts from sequential task streams while mitigating catastrophic forgetting and preserving base zero-shot generalization. Mainstream frameworks, such as ZiRa and DitHub, isolate task-specific knowledge by tuning modular class-specific LoRA adapters alongside frozen backbones.

Nevertheless, existing IVLOD paradigms fundamentally rely on a restrictive assumption: the Task-Incremental setting. During inference, models assume oracle task identities are provided a priori, allowing them to evaluate test images using constrained prompts restricted strictly to the categories of the given task. This reliance fails in realistic open-world deployments where task boundaries are unavailable, forcing detectors to localize and classify objects against an aggregated cross-task label space over a merged heterogeneous test set. The authors formalize this realistic setup as Task-Agnostic Incremental Vision-Language Object Detection (TA-IVLOD).

When transitioning from isolated single-task prompts to unified cross-task prompts, existing detectors suffer catastrophic performance collapse. This failure stems from two fundamental bottlenecks: first, cross-modal attention layers suffer severe attention dilution in the presence of extensive textual distractors, failing to route visual features to target semantic tokens; second, class-specific LoRA experts trained on disjoint tasks exhibit divergent optimization trajectories and low inter-task directional similarity. Merging them via static arithmetic averaging inevitably triggers severe parameter conflicts and representational collapse. Core idea: inject stochastically sampled historical classes and synthetic neutral noise into training prompts to build cross-modal distraction robustness, and dynamically aggregate class-specific LoRA experts at test time via online feature distribution modeling and Mahalanobis distance routing.

Method

Overall Architecture

The proposed TADA (Task-Agnostic Distribution-Aware Adaptation) framework is built upon the open-vocabulary detector Grounding DINO. It decouples the adaptation pipeline into training-time textual robustness enhancement and test-time visual parameter routing. During training, Stochastic Prompt Augmentation (SPA) dynamically constructs varied-length prompts by appending sampled previous classes and synthetic random noise strings to the current task categories, training cross-attention layers to resist semantic clutter while online class-mean vectors and a shared covariance matrix are estimated in a streaming manner. During task-agnostic inference, Test-Time Distribution-Aware Fusion (TTDF) evaluates the Mahalanobis distance between pooled visual features and class distribution profiles, using a temperature-scaled Softmax to dynamically weight class-specific LoRA matrices before combining them with the shared projection matrix.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data<br/>Current Image + Current Task Categories C_cur"] --> B["Stochastic Prompt Augmentation SPA<br/>Appends Sampled Old Classes C_prev and Synthetic Noise C_noise"]
    B --> C["Grounding DINO Backbone<br/>Multimodal Cross-Attention Enhancement & Detection"]
    C --> D["Online Distribution Modeling<br/>Streaming Updates of Class Means and Shared Covariance"]
    D --> E["Test-Time Distribution-Aware Fusion TTDF<br/>Mahalanobis Distance between Image Feature and Class Means"]
    E --> F["Dynamic LoRA Expert Aggregation<br/>Softmax Reweighted A_c Matrices with Shared B Matrix"]
    F --> G["Task-Agnostic Detection Output<br/>Accurate Prediction under Unified Cross-Task Prompt"]

Key Designs

1. Stochastic Prompt Augmentation (SPA): proactive training-time distraction simulation

To bridge the severe discrepancy between sparse single-task training prompts and consolidated cross-task inference queries, SPA actively introduces semantic distractors during training. For current task categories \(\mathcal{C}_{cur}\), SPA forms the augmented prompt \(\mathcal{P}_{train} = \mathcal{C}_{cur} \cup \mathcal{C}'_{prev} \cup \mathcal{C}_{noise}\). The historical replay tokens \(\mathcal{C}'_{prev}\) are dynamically sampled from previously encountered categories, forcing cross-modal attention to discriminate target objects from prior knowledge within a single sentence. The synthetic noise tokens \(\mathcal{C}_{noise}\) consist of randomly generated variable-length non-semantic strings (e.g., "asdf", "ghjklk"). These random tokens act as neutral surrogates for unseen future classes and out-of-vocabulary distractors, ensuring variable textual density without introducing spurious semantics that harm open-vocabulary zero-shot generalization. All injected distractor tokens participate solely in cross-modal feature fusion and are strictly excluded from ground-truth Hungarian bipartite matching and loss computation.

2. Online Distribution Modeling: exemplar-free tracking of class statistics

To enable dynamic parameter weighting at test time without violating exemplar-free continual learning constraints, TADA estimates class feature distributions online during training. For each class \(c \in \mathcal{C}_{gt}\), the framework maintains a running class mean vector \(\boldsymbol{\mu}_c\) and sample counter \(t_c\), alongside a globally shared scatter matrix \(\mathbf{S}\). Given visual feature \(f(\mathbf{x})\), the running class mean is updated in a streaming fashion:

\[\boldsymbol{\mu}_c^{(t)} = \boldsymbol{\mu}_c^{(t-1)} + \frac{1}{t_c} \left( f(\mathbf{x}) - \boldsymbol{\mu}_c^{(t-1)} \right)\]

Simultaneously, the shared scatter matrix \(\mathbf{S}\) accumulates outer products centered around updated means, normalized by total instances across all tasks to yield a shared covariance matrix \(\boldsymbol{\Sigma} = \frac{1}{\sum_c t_c} \mathbf{S}\). This online tracking requires minimal computational overhead and completely avoids caching raw images from past tasks.

3. Test-Time Distribution-Aware Fusion (TTDF): Mahalanobis distance-based dynamic expert routing

Rather than statically averaging class-specific LoRA adapters as in DitHub (\(\frac{1}{|\mathcal{P}|}\sum_{c} \mathbf{A}_c\))—which triggers catastrophic parameter conflicts across divergent tasks—TTDF dynamically weights adapters conditioned on the input visual distribution. For a test image \(\mathbf{x}\), TTDF extracts its pooled visual representation \(f(\mathbf{x})\) and computes the Mahalanobis distance to each learned class mean \(\boldsymbol{\mu}_c\):

\[d_c(\mathbf{x}) = \left( f(\mathbf{x}) - \boldsymbol{\mu}_c \right) \boldsymbol{\Sigma}^{-1} \left( f(\mathbf{x}) - \boldsymbol{\mu}_c \right)^\top\]

Unlike Euclidean distance which assumes isotropic variance, the Mahalanobis distance explicitly accounts for feature scale discrepancies and inter-channel correlations via the inverse covariance matrix \(\boldsymbol{\Sigma}^{-1}\), suppressing noisy feature dimensions. The distances are converted into normalized routing weights using a temperature-scaled Softmax: \(w_c(\mathbf{x}) = \frac{\exp(-d_c(\mathbf{x})/\tau)}{\sum_{j \in \mathcal{P}_{TA}} \exp(-d_j(\mathbf{x})/\tau)}\). The final adapted weights are assembled dynamically:

\[\mathbf{W} = \mathbf{W}_0 + \mathbf{B} \left( \sum_{c \in \mathcal{P}_{TA}} w_c(\mathbf{x}) \mathbf{A}_c \right)\]

This mechanism ensures that LoRA experts aligned with the visual content of the test image dominate the parameter space, while unrelated task experts are suppressed.

Loss & Training

The base Grounding DINO Swin-Tiny backbone remains entirely frozen throughout continual learning. Only the class-specific LoRA matrices \(\mathbf{A}_c\) and the shared projection matrix \(\mathbf{B}\) receive gradient updates. Training optimizes the standard Grounding DINO loss formulation, including binary focal loss for classification, L1 loss, and GIoU loss for bounding box regression. Injected distractor tokens are masked out of the target bipartite matching, ensuring that classification supervision remains focused on ground-truth instances present in the image.

Key Experimental Results

Main Results

Experiments are conducted on the ODinW-13 benchmark, comprising 13 diverse domains (e.g., Pascal VOC, Thermal, Aerial Maritime Drones, Aquarium) learned sequentially. Models are evaluated after learning all 13 tasks on the unified test set to compute average mAP (Avg), alongside zero-shot mAP on MS COCO (Zcoco).

Table 1: Quantitative comparison of mAP on ODinW-13 under the TA-IVLOD setting (from Table 1 in the paper)

Method	Zcoco	Avg	Aerial (Ae)	Aquarium (Aq)	Cottontail (Co)	EgoHands (Eg)	Mushroom (Mu)	Package (Pa)	PascalVOC (Pv)	Pistol (Pi)	Pothole (Po)	Raccoon (Ra)	Shellfish (Sh)	Thermal (Th)	Vehicles (Ve)
Grounding DINO (0-shot)	48.4	24.0	15.9	11.5	16.3	28.1	21.0	5.2	46.7	36.4	2.4	0.0	20.4	55.3	53.0
TFA	42.7	22.0	15.0	10.1	16.9	25.9	22.2	4.1	43.6	29.5	1.6	0.0	16.2	50.7	50.4
iDETR	37.4	34.0	30.1	30.7	44.1	40.6	31.6	7.0	58.9	37.0	10.8	0.2	30.3	63.5	57.4
ZiRa	45.1	35.0	28.5	31.4	46.5	34.6	35.8	4.6	59.0	37.6	15.5	0.1	32.7	65.7	62.4
DitHub	46.8	32.5	23.8	16.8	38.2	30.6	37.5	8.5	63.2	40.2	3.8	0.0	30.2	67.9	62.2
TADA (Ours)	47.2	64.5	43.4	50.3	82.0	70.6	51.0	89.0	69.3	68.6	53.3	61.4	56.2	71.4	71.3

Under the standard IVLOD setting with oracle task priors, TADA also establishes state-of-the-art accuracy (from Table 2 in the paper): - DitHub: Avg mAP 66.3, Zcoco 46.8 - TADA (Ours): Avg mAP 67.5 (+1.2), Zcoco 47.2

Ablation Study

Table 2: Ablation of TTDF and SPA components and augmentation choices (from Tables 4 and 5 in the paper)

Configuration / Strategy	TTDF	SPA	Augmentation Strategy	Zcoco	AvgTA (mAP)
Baseline (DitHub)	-	-	-	46.8	32.5
Add TTDF only	✓	-	-	47.4	45.9
Learned classes only	✓	✓	Replay seen classes	47.0	57.2
Synthetic noise only	✓	✓	Inject random strings	47.4	60.3
External COCO classes	✓	✓	Inject COCO class names	46.0	58.2
Full Model (TADA)	✓	✓	Learned classes + Synthetic noise	47.2	64.5

Table 3: Comparison of distance metrics in TTDF (from Table 6 in the paper)

Distance Metric	Zcoco	AvgTA (mAP)	Description
Euclidean Distance	45.9	51.3	Isotropic assumption vulnerable to high-variance noisy dimensions
Mahalanobis Distance	47.2	64.5	Normalizes channel variance and accounts for cross-feature covariance (+13.2)

Key Findings

• In the task-agnostic setting without task priors, previous methods experience severe performance collapse due to unconstrained negative samples and attention dilution (DitHub drops from 66.3 in IVLOD to 32.5 in TA-IVLOD; Raccoon drops to 0.0 mAP). TADA achieves 64.5 mAP, demonstrating exceptional resilience against cross-task distractors.
• TTDF provides an initial gain of +13.4 mAP by dynamically routing parameters, while adding SPA contributes an additional +18.6 mAP, proving the necessity of joint text-side regularization and visual-side dynamic aggregation.
• Synthetic noise tokens outperform real external category names (such as COCO classes) as distractors. Introducing real semantic categories confuses the detector into treating valid concepts as background, dropping COCO zero-shot mAP to 46.0; meaningless synthetic strings avoid semantic interference while fortifying cross-modal attention.

Highlights & Insights

• Formulation of TA-IVLOD: Exposes the artificial simplicity of evaluating incremental vision-language detectors with oracle task priors, bridging the gap between benchmark evaluations and unconstrained open-world deployment.
• Semantic-neutral regularization via random noise: Leveraging non-semantic random character strings in textual prompts is a simple yet elegant mechanism that trains cross-attention layers to handle variable-length distractors without impairing base zero-shot representations.
• Exemplar-free streaming covariance routing: Dynamically aggregating modular LoRA adapters using online-estimated Mahalanobis distance establishes an effective paradigm for multi-expert model merging without caching raw training images.

Limitations & Future Work

• Inference latency overhead: Dynamic test-time parameter aggregation requires calculating Mahalanobis distances and weighted LoRA additions for each input sample, introducing slight computational overhead compared to static offline model merging.
• Coarse granularity of image-level pooling: TTDF computes routing weights using globally pooled image features. When an image contains multiple objects from different historical tasks, global pooling may produce mixed representations; extending distribution-aware routing to region-proposal or instance-level granularity represents a valuable future direction.

Related Work & Insights

• vs DitHub: DitHub pioneered modular class-specific LoRA expansion for IVLOD but relied on oracle task prompts or static arithmetic averaging during inference, resulting in severe parameter conflicts under unified cross-task prompts (32.5 mAP). TADA builds upon this modular foundation, resolving parameter conflicts via SPA and TTDF to gain +32.0 mAP in TA-IVLOD.
• vs ZiRa: ZiRa introduced re-parameterization and zero-interference loss to protect base classes, yet heavily depends on task boundaries and suffers attention dilution under long prompts. TADA outperforms ZiRa across downstream adaptation and base class retention without requiring distillation losses.
• vs Conventional Incremental Object Detection (IOD): Traditional IOD methods (e.g., iDETR, GCD) operate primarily on closed vocabularies within single datasets. TADA demonstrates strong cross-domain adaptability across 13 diverse domains while preserving a 47.2 COCO zero-shot mAP.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pioneers the realistic TA-IVLOD formulation; proposed stochastic noise augmentation and Mahalanobis parameter routing are elegant and effective.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across ODinW-13 in both full and few-shot regimes, standard IVLOD, traditional IOD benchmarks, and extensive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation supported by intuitive visualizations of cross-attention heatmaps and LoRA similarity matrices.
• Value: ⭐⭐⭐⭐☆ Highly practical framework for deploying continually adaptive open-vocabulary detectors in real-world environments.

Related Papers

• [CVPR 2026] Distribution-Aligned Multimodal Fusion for Robust Object Detection
• [CVPR 2026] Parameterized Prompt for Incremental Object Detection
• [ECCV 2026] DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images
• [ECCV 2026] VLOD-TTA: Test-Time Adaptation of Vision-Language Object Detectors
• [ECCV 2026] Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection