DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images

Conference: ECCV2026
Paper: ECCV page
Area: Object Detection
Keywords: UAV imagery object detection / parameter-efficient fine-tuning / vision-language model / foreground-aware adapter / background suppression

TL;DR

Building on a frozen GroundingDINO with only 1.54M trainable parameters (5.6% of the 27.5M used by full fine-tuning), DroneFINE combines two domain-aware modules — the foreground-aware dynamic multi-path convolution adapter HyperAdapter and the background-vocabulary gated query re-ranking module SemanticGate — and reaches 38.4 mAP / 61.2 mAP50 on VisDrone and 26.5 mAP / 43.6 mAP50 on UAVDT, matching (and on UAVDT surpassing) full fine-tuning while preserving generalization on COCO.

Background & Motivation

Drones increasingly rely on object detection as the basic perception layer for inspection, emergency rescue and logistics delivery, and vision-language model (VLM) detectors such as GLIP, GroundingDINO and YOLO-World should in principle suit the open and dynamic environments UAVs operate in, thanks to their image-text alignment and open-vocabulary capability. The obstacle is the large domain gap between VLM pre-training data and aerial imagery: VLM visual priors come from foreground-dominant, ground-level datasets such as COCO, Objects365, V3Det and GRIT, whereas drone images are bird's-eye-view, background-dominant, and filled with small, densely packed objects. Under direct zero-shot transfer the gap is striking — categories like "people" and "awning-tricycle" reach only about 2%–20% mAP50 on VisDrone, and the overall zero-shot score is 16.5 mAP. The alternative route, full fine-tuning or retraining (SPAR, LAE-DINO), does improve accuracy, but it updates every parameter and depends on large-scale aerial data and compute; on small in-domain datasets it overfits easily and even forgets the VLM's valuable pre-trained knowledge, which defeats the whole purpose of using a VLM.

The natural move is therefore to apply general PEFT methods (LoRA, Adapter, VPT, Mona, CoOp) directly. The authors' empirical analysis identifies two concrete reasons why these methods fail in the UAV domain. First, the rank is insufficient: decomposing the FFN weight matrices with SVD shows that maintaining a low approximation error on aerial data requires a very high rank, especially in deeper layers, so restricting adaptation to a low-rank linear space cannot reconstruct the complex spatial detail of aerial objects. Second, the architecture is static: even when convolutions are introduced to supply a spatial inductive bias, as in Mona, a globally shared set of filters still applies one uniform feature-extraction paradigm to the entire image and cannot cope with the severe intra-domain variation in scale, density and appearance; without dynamic routing, representational capacity is spent on redundant background. This is compounded by the background-dominant nature of aerial imagery, where the model's finite attention budget is drawn toward the background while foreground small objects have already collapsed into sparse pixel clusters that are barely distinguishable from noise — together these cause VLMs to fail mainly on small, densely distributed objects.

The angle taken here is not to stack yet another domain-specific detection module, but to do two complementary things while keeping the VLM backbone frozen: turn the adapter's convolution kernels into a high-rank representation generated dynamically from foreground content, and extend the VLM's text branch from "describing targets only" to "explicitly describing the background", using background semantics to suppress background responses. Core idea: on top of GroundingDINO, introduce two domain-aware PEFT modules — HyperAdapter, which aggregates global foreground semantics with learnable foreground queries and generates multi-path depthwise separable kernels through a layer-shared hypernetwork, and SemanticGate, which re-ranks the language-guided query selection with the text embeddings of 20 background words and keeps foreground/background text features separable with a token-level contrastive loss — reaching full fine-tuning performance with only 1.54M trainable parameters.

Method

Overall Architecture

The method is built on a pre-trained GroundingDINO, with both the visual and the text backbone frozen throughout; only the inserted adaptation modules, the decoder and the detection head are trained. It can be read as two cooperating routes. On the visual route, a HyperAdapter is spliced after the attention and MLP layers of every Transformer block: M learnable foreground queries first aggregate the sample's global foreground semantics, a shared hypernetwork maps them into dynamic convolution kernels, those kernels run multi-path depthwise separable convolutions over low-rank-projected features, and the result is projected back to the original dimension and added back to the backbone as a residual — raising representational rank and spatial selectivity at almost no parameter cost. On the text–query route, an offline pipeline first builds a vocabulary of 20 background words and pre-computes their text embeddings; at runtime the background embeddings are matched against image features to obtain a background score per candidate, which is fed together with the original foreground score into the three gates of SemanticGate (protection gate, inhibition gate, candidate adjustment), re-ranking the top-K queries produced by GroundingDINO's language-guided query selection so that more of the selected queries land on real foreground. The trainable decoder and detection head then output the boxes. Both routes pursue the same goal: redirecting the limited attention and representational capacity from background to foreground.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image + text prompt<br/>visual/text backbone frozen"] --> B["Foreground-Aware Query Aggregation"]
    B --> C["Shared-Hypernetwork Multi-Path Convolution Generation"]
    C --> D["Adapted visual features<br/>residual back to backbone"]
    E["Background Vocabulary Construction<br/>sampling→LLM→DINO-X→manual review"] --> F["SemanticGate Gated Query Re-ranking"]
    D --> F
    F --> G["Trainable decoder and detection head"]
    G --> H["Detected boxes"]

Key Designs

1. Foreground-Aware Query Aggregation: the kernel is decided by the foreground in the image

The trouble with a static adapter is that it treats the whole image alike: one set of filters must work on sparse pedestrian pixel clusters and on large stretches of road background at the same time, so its capacity is diluted. HyperAdapter's answer is to maintain \(M\) learnable foreground queries and run cross-attention between them and the flattened feature map. The queries themselves are optimized over the entire training set and therefore encode a dataset-level, cross-image prior ("what foreground roughly looks like from a drone's viewpoint"), while their attention distribution on a single feature map describes where the foreground is in that image — exactly the combination of cross-sample commonality and per-sample individuality that is needed. After normalizing the attention weights over the spatial dimension, a weighted sum of the features followed by an average over the \(M\) queries yields the sample's global foreground feature:

\[A=\operatorname{softmax}\!\left(q\,{x^{\mathrm{flat}}}^{\top}\right)\in\mathbb{R}^{M\times HW},\qquad z=\frac{1}{M}\sum_{j=1}^{M}\sum_{k=1}^{HW}A_{j,k}\,x^{\mathrm{flat}}_{k}\]

Note that this aggregation is implicit rather than an explicit localization: plugging ESOD's ObjSeeker (which finds foreground via a density-map prediction head) into the Adapter gives only a marginal gain over the baseline (37.3/60.0) and stays clearly below HyperAdapter (38.1/61.1), and replacing HyperAdapter's attention scores with density-map scores brings no improvement at all. Cross-attention already suffices to learn what features should generate the kernels, whereas an auxiliary task head would have to be paid for at every adapter layer and would undercut the whole point of PEFT.

2. Shared-Hypernetwork Multi-Path Convolution Generation: a high-rank representation from very few generating parameters

Dynamically generated convolution kernels are notorious for unstable training and parameter blow-up. HyperAdapter answers with two concrete choices. First, a single hypernetwork \(H\) is shared across all adapter layers rather than one generator per layer, which saves parameters and lets gradients from different depths converge on the same generator, stabilizing training. Second, the generation target is multi-path, with parallel 1×1, 3×3 and 5×5 depthwise separable branches that cover the wide range of object scales in aerial images and provide richer gradient feedback to the generator. To keep the generated parameter count minimal, the hidden dimension of \(H\) is compressed, depthwise separable convolutions are used, and parameters are shared across groups. The full data flow is: the input is first scale-scaled as in Mona to give \(\tilde{x}\), projected into the low-rank space by \(W_{down}\) and reshaped, convolved with the dynamic kernels, activated by GELU, projected back by \(W_{up}\) and added to the input:

\[x_{out}=x+W_{up}\Big(\mathrm{GELU}\big(\mathrm{DynConv}(\,W_{down}\tilde{x};\,H(z)\,)\big)\Big)\]

Why this beats simply raising the rank is clear from the ablations: increasing the rank from 64 to 96 drops performance from 38.4/61.2 to 37.9/60.9, scaling LoRA to rank 128 (2.26M parameters) still sits at 37.1/59.6, and Mona at r=128 (2.97M parameters) reaches only 37.9/60.6 — comparable or even double the parameter count of HyperAdapter's 1.54M (38.1/61.1) yet less accurate. The bottleneck is not the number of parameters but whether the kernels are data-dependent.

3. Background Vocabulary Construction: turning "what to ignore" into a grounded, reusable text prior

Detection protocols usually care only about target categories, which throws away the VLM's open-vocabulary ability: if the model knows "road", "building" and "vegetation", it can describe the background with them. Feeding an arbitrary word list into the similarity computation is meaningless, though — the paper's random-vocabulary control shows that growing a random list from 5 to 20 words yields no consistent gain. The vocabulary therefore has to be constructed deliberately, and the authors use a four-stage pipeline: sample representative aerial images and analyse the background distribution to identify three dominant contexts — streetscapes, natural terrains and atmospheric conditions; prompt GPT-4 to generate a pool of candidate keywords; since not every textual concept is visually salient from a bird's-eye view, ground the candidates on real aerial images with DINO-X and discard those that cannot be detected; finally conduct a strict manual review to remove terms with unstable detection results or ambiguous referents, ending with 20 finalized words (e.g. "road", "building"). Once fixed, their text embeddings are pre-computed, so inference incurs no extra cost — which also explains why adding SemanticGate leaves the total trainable parameter count essentially unchanged.

4. SemanticGate Gated Query Re-ranking: select queries with the background score instead of inflating the foreground score

GroundingDINO's language-guided query selection is essentially an anchor-box-like mechanism that dictates where the model places its attention. Aerial imagery exhibits an asymmetric image-text alignment: foreground features struggle to align with text prompts because of the domain gap, while the dominant background elements remain highly detectable and respond strongly, so the raw foreground similarity has little discriminative power in background regions. SemanticGate therefore stops ignoring the background passively and instead identifies and suppresses it actively: the maximum similarity between image features and background-word embeddings, \(s_{bg}\), is computed and fed together with the maximum original foreground similarity \(s_o\) into three cooperating gates that jointly determine how a candidate score is adjusted —

\[ \begin{aligned} g_{pg}&=\sigma\!\left(w_{pg}^{bg}s_{bg}+w_{pg}^{o}s_o+b_{pg}\right),\qquad g_{ig}=\sigma\!\left(w_{ig}^{bg}s_{bg}+w_{ig}^{o}s_o+b_{ig}\right)\\ a_{adj}&=\alpha\cdot\tanh\!\left(w_{ca}^{bg}s_{bg}+w_{ca}^{o}s_o+b_{ca}\right) \end{aligned} \]

The protection gate (PG) reads the foreground confidence and sets the retention ratio of the original score (when the foreground score is high, \(g_{pg}\) approaches 1 and the potential target query is safely "protected"); the inhibition gate (IG) approaches 1 on background-dominated candidates with a high background score and a low foreground score, switching suppression on; the candidate adjustment module (CA) produces an adjustment magnitude bounded by tanh and scaled by the hyper-parameter \(\alpha\). The adjusted score \(S_{adj}\) is built on the foreground score and corrected by \(g_{ig}\) and \(a_{adj}\) (⚠️ the exact combination of the three terms in Eq. (11) is garbled in the cached text — refer to the original paper), and is used to re-rank the top-K queries. The gates are only meaningful if \(s_{bg}\) and \(s_o\) are themselves separable, so an InfoNCE-based token-level contrastive loss is applied to the text embeddings used for matching, pushing tokens of different semantics such as "road" and "car" apart in the embedding space.

The effect is quantified concretely: on VisDrone, the GT-box IoU coverage of the top-20 queries improves on 19.3% of images and degrades on 6.0%, while [email protected] improves on 6.8% and degrades on 2.0%. Most tellingly, the foreground similarity actually decreases slightly on 23.9% of images — so the gain does not come from simply boosting foreground scores but from a genuine re-ranking driven by background scores. A single-gate variant loses 0.7 mAP50 on VisDrone, supporting the multi-gate design.

Loss & Training

The training objective has two parts: the original detection loss of GroundingDINO, and the added token-level contrastive loss \(\mathcal{L}_{con}\), which concatenates all text tokens used for image-text matching (foreground prompts and background words) and applies token-level InfoNCE to push semantically different tokens apart, ensuring that background suppression operates on high-quality separable features. Implementation follows MMDetection's GroundingDINO, and all fine-tuning strategies share a low-rank dimension of 64 for a fair comparison; both the text and the visual backbone are frozen, and only the decoder, the detection head and the inserted adaptation modules are trained, with DroneFINE-T using 1.54M trainable parameters relative to the baseline (full fine-tuning uses 27.5M). The main text does not report learning rate, epoch count or other optimization details (they may appear in the supplementary material), so they are not speculated here.

Key Experimental Results

Main Results

DroneFINE-T is compared with representative PEFT methods under identical settings on VisDrone and UAVDT (excerpted from Table 1; "trainable parameters" is the increment relative to the baseline):

Method	Trainable Params	VisDrone mAP	VisDrone mAP50	UAVDT mAP	UAVDT mAP50
Zero-shot	–	16.5	26.9	9.5	18.5
Baseline (decoder + head only)	0	37.1	59.6	24.7	40.9
Full Fine-tuning	27.5M	38.4	61.3	22.4	38.6
Adapter	1.14M	37.4	60.1	22.5	39.3
LoRA	1.13M	36.7	59.3	25.0	40.8
NormTuning	0.025M	37.1	59.6	22.7	40.1
Mona	1.4M	37.9	60.5	22.2	37.6
VPT	0.08M	37.1	59.6	22.3	37.8
CoOp	0.012M	37.3	60.0	24.9	41.4
Shine	0.02M	37.0	60.1	24.3	42.4
DroneFINE-T (Ours)	1.54M	38.4	61.2	26.5	43.6

Comparison with UAV detectors and other VLM detectors (excerpted from Table 2; the other VLMs are pre-trained on LAE-1M, unlike this work):

Category	Method	Backbone	VisDrone mAP / mAP50	UAVDT mAP / mAP50
Traditional UAV models	ClusDet (ICCV'19)	ResNet-50	26.7 / 50.6	13.7 / 26.5
Traditional UAV models	UFPMP-Det (AAAI'22)	ResNet-50	36.6 / 62.4	24.6 / 38.7
Traditional UAV models	QueryDet (CVPR'22)	ResNet-50	28.3 / 48.1	–
Traditional UAV models	CEASC (CVPR'23)	ResNet-50	28.7 / 50.7	17.1 / 30.9
Recent SOTA	RemDet-X / RemDet-L (AAAI'25)	YOLOv8X	40.0 / 61.9	20.6 / 34.5
Recent SOTA	ESOD (TIP'24)	YOLOv5	37.9 / 62.3	23.6 / 47.6
Recent SOTA	Dome-DETR (ACM MM'25)	HGNetv2L	39.0 / 61.1	–
VLM	YOLO-World (CVPR'24)	YOLOv8L	– / 55.3	– / 35.8
VLM	RT-OVAD (arXiv'25)	ResNet-50	– / 64.6	– / 40.5
VLM	LAE-DINO (AAAI'25)	Swin-T	– / 56.4	– / 36.5
Ours	DroneFINE-T	Swin-T	38.4 / 61.2	26.5 / 43.6
Ours	DroneFINE-B	Swin-B	38.8 / 61.9	–
Ours	DroneFINE-L	Swin-L	42.3 / 65.8	–

Ablation Study

Config	VisDrone mAP / mAP50	UAVDT mAP / mAP50	Note
Baseline	37.1 / 59.6	24.7 / 40.9	decoder + head only
HyperAdapter (full)	38.1 / 61.1	26.1 / 41.9	added alone; +1.5 / +1.0 mAP50 over baseline
HyperAdapter w/o foreground awareness	37.8 / 60.2	24.9 / 41.0	−0.9 mAP50 on both datasets
SemanticGate (full)	37.9 / 60.8	26.0 / 42.0	added alone; +1.2 / +1.1 mAP50 over baseline
SemanticGate w/o contrastive loss	37.5 / 60.4	25.2 / 42.1	both mAP and mAP50 drop on VisDrone; UAVDT mAP50 rises by 0.1
Mona + CoOp	37.5 / 60.3	24.5 / 41.7	naive visual-side + text-side combination, far behind the two modules together
DroneFINE-T (full model)	38.4 / 61.2	26.5 / 43.6	both modules; +1.3 / +2.7 mAP over baseline

Key Findings

• Background semantics matter, vocabulary size does not: the full 20-word vocabulary reaches 37.9/60.8, i.e. +0.1/+0.5 over the best single-category vocabulary (natural terrains, 8 words, 37.8/60.3) and +1.0/+1.4 over a size-matched Random-20 vocabulary (36.9/59.4), while growing a random list from 5 to 20 words yields no consistent gain. Within single categories, atmospheric conditions is the weakest (37.2/59.6), suggesting the complementarity of the three background types is what drives the improvement.
• Rank is not the bottleneck — dynamics is: rank 64 is optimal (38.4/61.2), while going down to 32 (37.3/60.0) or up to 96 (37.9/60.9) both hurt; LoRA at r=128 (2.26M) and Mona at r=128 (2.97M) only reach 37.1/59.6 and 37.9/60.6, both below HyperAdapter's 1.54M (38.1/61.1). This directly supports the authors' diagnosis that low-rank structure limits expressiveness.
• In-domain overfitting is the real risk of PEFT, and this method is stable on both sides: on UAVDT, full fine-tuning drops to 22.4/38.6, below the baseline's 24.7/40.9 (highly similar video frames cause overfitting), whereas text-side PEFT regularizes better with few parameters (CoOp 24.9/41.4, Shine 24.3/42.4); this work matches full fine-tuning on VisDrone (38.4 vs 38.4) and beats it by 4.1 mAP on UAVDT (26.5).
• Strong generalization and anti-forgetting: on COCO the method reaches 44.9 mAP, above full fine-tuning (42.5) and Mona (41.9), showing that in-domain adaptation does not sacrifice pre-trained knowledge. The inference cost is small: on a single 2080 Ti the speed drops from the original model's 3.22 FPS to 2.79 FPS, only 0.43 FPS slower.
• Qualitative results: the baseline misclassifies dumpsters as cars (an incorrect association of their aerial appearance with the pre-learned "car" concept), while this method detects the distant truck and the rare "bicycle" category; attention maps confirm that the foreground-aware module indeed focuses on vehicles and pedestrians.
• Boundaries of the SOTA comparison: DroneFINE-L (Swin-L) at 42.3 mAP exceeds the previous best UAV detector (RemDet-X, 40.0 mAP) by 2.3 points, and its mAP50 is 3.4 points above UFPMP-Det's 62.4; however, RT-OVAD already reaches 64.6 mAP50 in the same table, only 1.2 below 65.8, and it uses large-scale LAE-1M aerial pre-training, so the data conditions are not equivalent. On UAVDT, 26.5 mAP surpasses ESOD by 2.9 while mAP50 is slightly lower, which the authors explain by ESOD re-cropping foreground regions and favouring foreground detection over precise localization at high IoU.

Highlights & Insights

• Treating background as a usable supervisory signal rather than noise: most domain-adaptation work filters background out, whereas this paper describes it explicitly with 20 groundable background words and re-ranks attention with them. The benefit is cheap (embeddings are pre-computed, so trainable parameters barely change) and reusable — any domain with a stable background distribution can build such a vocabulary the same way.
• The source of the gain is pinned down cleanly: on 23.9% of images the foreground similarity actually decreases while GT coverage still improves on 19.3% of images, a pair of statistics that rules out the mundane explanation of "just inflating foreground scores" and shows the gates perform genuine re-ranking — a solid evidence-driven analysis.
• A parameter-thrifty combination for hypernetworks: one generator shared across layers + compressed hidden dimensions + depthwise separability + group-shared parameters + multi-path branches to restore gradients. This replaces the usual one-generator-per-layer design and can be transferred directly to any setting that wants a spatial inductive bias inside a low-rank adapter.
• Diagnose before designing: SVD rank-energy curves show that low rank is insufficient, and feature-similarity curves show the representation gap between LoRA and full fine-tuning; the modules are then designed around these measurements. The style itself is worth borrowing.
• Implicit foreground aggregation wins: no density-map head and no auxiliary loss — cross-attention between foreground queries and the feature map alone learns the foreground cues, while the explicit density-map alternative (ObjSeeker) performs worse, suggesting attention is already sufficient for aggregating global foreground semantics.

Limitations & Future Work

• Vocabulary construction is manual and not automated: from GPT-4 candidate generation and DINO-X grounding to manual review, the pipeline depends on human judgement, and moving to a new domain (night flight, infrared, dense high-rise areas) requires redoing it. No automatic construction or cross-domain vocabulary transfer is offered.
• Vocabulary size and coverage are under-explored: only 20 words over three background types survive; the paper shows that more random words do not help, but never answers whether covering more background types (shadows, water reflections, moving non-vehicle objects) would keep improving.
• Per-gate contributions remain unclear: only single-gate vs multi-gate is compared (single gate loses 0.7 mAP50), with no ablation removing PG, IG or CA individually; the exact form of \(S_{adj}\) and the effect of \(\alpha\) are not developed (⚠️ that equation is garbled in the cached text — check the original paper).
• One base model and two datasets: all experiments use GroundingDINO on VisDrone/UAVDT, with no validation on other VLM detectors (e.g. YOLO-World) or other aerial datasets; the supplementary comparison details cannot be judged from the main text either.
• Watch the parameter accounting: the 1.54M figure is an increment relative to the baseline, and the baseline itself already trains the decoder and detection head; the "5.6% of full fine-tuning" claim compares against the 27.5M of full FT. The two scopes differ and should not be mixed when citing.
• Real-time use is still limited: 2.79 FPS on a single 2080 Ti is far from on-board deployment, and deployment-side optimizations such as quantization or distillation are not discussed.

Related Work & Insights

• vs LoRA / Adapter: they update a fixed low-rank, fixed-structure parameter increment with no sample adaptivity; on spatially complex, dense-small-object VisDrone, LoRA even falls below the baseline (36.7 vs 37.1) and on UAVDT it establishes no clear advantage. This work keeps the parameter thrift of low-rank projections but hands the kernels to a data-dependent generator.
• vs Mona: Mona also injects convolutions for a spatial inductive bias, but its kernels are static and globally shared; here the kernels become data-dependent, and the rank-scaling comparison shows Mona still loses at 2.97M parameters to HyperAdapter's 1.54M — the difference comes from dynamic routing, not capacity.
• vs SPAR / LAE-DINO: they take the full retraining route, needing large-scale aerial data and compute, and risk catastrophic forgetting on small in-domain datasets (here full fine-tuning drops to 22.4 mAP on UAVDT, below the baseline); this work matches full fine-tuning on VisDrone with 1.54M trainable parameters and is 2.4 mAP higher on COCO.
• vs ObjSeeker (ESOD): both aim to find foreground; ESOD localizes explicitly with a density-map prediction head, while this work aggregates implicitly through query attention and avoids attaching an auxiliary task head to every adapter layer, balancing accuracy against compute.

Rating

• Novelty: ⭐⭐⭐⭐ Combining foreground-driven dynamic kernel generation with background-vocabulary gating inside PEFT is new, though each part traces back to hypernetwork adapters and prompt learning.
• Experimental Thoroughness: ⭐⭐⭐⭐ Two datasets and five ablation groups (modules, rank, vocabulary, query statistics, anti-forgetting and speed), but no per-gate ablation and no cross-backbone validation.
• Writing Quality: ⭐⭐⭐⭐ Motivation is backed by SVD rank analysis, feature similarity and query statistics, and the structure is clear; the gating equations are not expressed cleanly and some details need checking against the original.
• Value: ⭐⭐⭐⭐ Reaching full fine-tuning quality with 5.6% of the parameters and better anti-forgetting makes this directly useful for practitioners deploying VLMs on drones.