DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

Conference: ECCV2026
Paper: ECCV Paper
Authors: Gautam Rajendrakumar Gare, Neehar Peri, Matvei Popov, Shruti Jain, John Galeotti, Deva Ramanan
Area: Object Detection
Keywords: few-shot object detection, in-context learning, prompt optimization, visual error feedback, confidence calibration

TL;DR

DetPO converts a few annotated images into repeatedly validated textual class definitions, using false positives to narrow concepts and false negatives to broaden them, improving few-shot detection with frozen multimodal large language models and optionally refining detection rankings through per-box VQA re-scoring.

Background & Motivation

Generalist multimodal large language models (MLLMs) can generate bounding boxes directly, but localization on familiar benchmarks does not guarantee an understanding of new annotation concepts. RF20-VL covers aerial imagery, medical imaging, industrial defects, and actions, where targets may be defined by material properties, local anomalies, or scene relationships. For example, two people striking a ball can have similar poses while belonging to different classes depending on whether they are near the net or behind the service line. Class names alone cannot fully express such boundaries, while fine-tuning requires weight access and additional training resources. For API-only models, adaptation must operate through inputs and outputs.

Putting annotated examples directly into the context seems natural, but is not a reliable solution. In Table 1, Qwen3-VL 8B achieves 11.4 mAP with class names and instructions, falling to 7.0 when image examples are added. Qwen2.5-VL 7B similarly falls from 6.2 to 1.8; however, Gemini 3 Pro improves from 23.0 to 23.9 in that table, so the effect is not universally negative. The authors hypothesize that rigid post-training prompt formats limit the use of additional visual context, but do not directly establish this training-level causal explanation. The practical question is how to present the information already contained in annotated examples in a form the target detector can use.

DetPO does not ask the detector to imitate examples during testing; it first uses those examples to discover a more accurate class definition. A scalar detection score can tell an optimizer that performance worsened, but not what the model confused; visual false positives and false negatives indicate how the concept boundary should move. Visual examples therefore support prompt discovery, while deployment retains only the optimized text and the image to be detected. Core Idea: turn few-shot adaptation into textual concept revision grounded in visual errors, then handle class understanding and per-box confidence estimation separately.

Method

Overall Architecture

Inputs comprise class names, a few annotated training images, and a few-shot validation set for selecting candidate prompts; outputs are classes, bounding boxes, and confidence scores on test images. The target model detects objects, while the critique model reads visual evidence and revises class descriptions; experiments generally use the same MLLM for both roles, rather than training two new networks. Contrastive prompt refinement is followed by validation selection and self-reported scores, producing a deployable detector, with optional per-box VQA re-scoring afterward. A text-only prompt means that training example images are no longer attached; the test image itself remains a multimodal input.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Train["Few-shot training images<br/>Class names and ground-truth boxes"] --> Refine["Contrastive prompt refinement"]
    Refine -->|Training false positives and false negatives| Refine
    Refine --> Select["Validation selection and<br/>self-reported scores"]
    Valid["Few-shot validation set"] -->|Prompt selection only| Select
    Test["Image to detect"] -->|Deployment input| Select
    Select -->|Optional| VQA["Per-box VQA re-scoring"]
    Select -->|Default direct output| Output["Classes, boxes, and confidence"]
    VQA --> Output

Training-image errors are feedback for adaptation, and validation images only select prompts; neither should be treated as a required test-time input. With optional VQA, the scorer revisits test images and predicted boxes but does not access test annotations. The method changes natural-language instructions and prediction rankings, not the weights of the visual encoder, language model, or detection head.

Key Designs

1. Contrastive prompt refinement: narrow definitions with false positives and broaden them with false negatives

Initialization is more than expanding a class name. For each class, the model first examines ground-truth training instances and summarizes their shared visual properties. It then contrasts that class with ground-truth instances from other classes, retaining distinguishing attributes rather than merely listing common colors, shapes, or textures. Using this initial description, the target model detects objects in training images, and ground-truth boxes identify false positives and false negatives. This matters because a plausible description is not necessarily one that produces correct detections. The contrastive information comes from the actual task's class boundaries, rather than independently imagined world knowledge.

Each iteration selects feedback from the most severe errors; the paper gives high confidence for false positives and low IoU for false negatives as examples. The critique model sees full images with differently colored boxes marking the reference ground-truth instance and the error, allowing scene relationships to inform revision instead of relying only on cropped objects. For a false positive, it first compares differences from a reference positive, then revises the definition to exclude the incorrect instance. For a false negative, it first identifies similarities to the reference, then broadens the definition to include the missed valid instance. The revised prompt is tested on the training set again, repeating until training performance converges or the maximum refinement budget is reached. The available main text does not specify the error sample count \(K\), matching thresholds, or maximum iteration count, so these cannot be filled in as fixed settings.

This process gives optimization a clearer direction than sending only numeric IoU feedback to a general-purpose prompt optimizer. When an occluded instance is missed, visual feedback can reveal an overly restrictive implicit requirement for complete visibility. When a similar action is incorrectly detected, revision can add location or action-phase constraints instead of repeatedly paraphrasing synonyms. Figure 2 illustrates removing a black-jersey restriction and adding an actively-jumping condition, showing that revision can both remove and introduce attributes. These edits adjust concept boundaries rather than coordinate regression rules; they do not directly guarantee more precise boxes.

2. Validation selection and self-reported scores: select generalizable descriptions and make detections rankable

Increasingly detailed descriptions can memorize incidental features of a few training examples, so the last iteration should not automatically become the deployed prompt. After optimization, the method generates multiple candidate refinements and evaluates them alongside the initial prompt on a few-shot validation set. The best validation prompt is used for testing; retaining the initial prompt also allows a fallback to a simpler definition. This selects discrete text candidates rather than applying gradient descent to prompt embeddings or searching descriptions against test labels. The main text does not give the validation sample count or candidate count, which still require the appendix or code for reproduction.

A separate issue is that MLLMs normally return boxes without the per-box confidence scores typical of specialist detectors. Assigning every baseline box a score of 1.0 removes ranking information that could separate reliable predictions from false positives. DetPO explicitly asks for a score alongside each box, producing boxes and confidences in the same detection generation. Unlike subsequent per-box questioning, this requires no additional model calls; nevertheless, a self-reported score is not automatically a calibrated probability of correctness. The optimization loop uses this inexpensive scoring to avoid paying for full per-box visual verification at every refinement step. Deployment can retain these scores directly; the Qwen3-VL result of 19.4 mAP uses this setting, not VQA post-processing.

3. Per-box VQA re-scoring: replace joint self-reporting with instance-specific visual evidence

Self-reported scores are generated jointly with all detections, without necessarily checking the visual evidence for each box independently. Optional post-processing draws one predicted box on the original image and asks whether it contains an instance of the specified class, requiring a Yes or No answer. Keeping the original image rather than only the box crop preserves information about actions, context, and neighboring objects. After evaluating each box independently, its new score is computed from the Yes and No token probabilities:

\[ s_{\mathrm{VQA}} = \frac{p(\mathrm{Yes})}{p(\mathrm{Yes}) + p(\mathrm{No})}. \]

The fraction is misaligned in the extracted cache; this expression follows the explicit normalized-Yes-probability description on page 7. It only re-estimates confidence for existing boxes, without moving coordinates or creating detections absent from the first stage. Independent inspection can lower the rank of false positives but can also suppress genuine objects and reduce recall. Each prediction requires an additional evaluation, so computation scales linearly with the number of predicted boxes; this improvement is not free. Because the Gemini 3 Pro API does not expose the required token probabilities, the paper uses Qwen3-VL 30B-A3B to re-score Gemini's boxes. Those final results therefore include a cross-model combination and should not be described as calibration performed by Gemini alone.

A Worked Example

Consider the volleyball classes Attack and Serve in Figure 3: both can depict an airborne person striking the ball. An initial prompt focused on pose can confuse a serve with an attack; adding more images of similar poses may not resolve the ambiguity. Contrastive prompt refinement compares the erroneous prediction with a genuine attack in full-scene context, revealing the net and service-line position as better distinguishing cues. The Attack description then emphasizes proximity to the net, and Serve emphasizes being behind the service line, before training images are re-evaluated for confusion. After validation selects an effective description, testing uses only the new image and selected instructions to produce candidate boxes and self-reported scores. With VQA enabled, each box receives another class-specific judgment in the full test image, moving low-scoring boxes down the detection ranking. This example explains information flow without inventing candidate counts or per-example AP that the figure does not report.

Loss & Training

DetPO has no loss over trainable parameters; class descriptions are optimized using few-shot detection performance as feedback. RF20-VL provides 10-shot training examples and rich annotation instructions per class, and the main text additionally uses a few-shot validation set to select candidate prompts. Test-time optimization means gradient-free adaptation to the target task, not using test labels or repeating prompt search for every test image. Prompt-discovery calls are generally paid once and the resulting text is reused; optional VQA continues to incur per-box calls for every test image. Figure 4 reports that gains begin to plateau around iteration 6 in most domains, which is an observation rather than a universal stopping hyperparameter. Appendices A, C, and D cited by the paper are absent from the supplied cache, so detailed model settings, pseudocode, and complete prompt templates could not be verified.

Key Experimental Results

Main Results

RF20-VL contains 20 datasets and uses standard COCO detection mAP; All is an average across datasets, not a simple average readers should reconstruct from the displayed domain columns. The following selection comes from Table 2 on page 9 and Table 3 on page 10; every result is RF20-VL All mAP. Class names, instructions, and images describe prompt information; DetPO uses few-shot images for prompt discovery without attaching those examples directly at test time.

Model	Config	All mAP	Source
GroundingDINO	Class names	16.8	Table 2
LLMDet	Class names	17.2	Table 2
Qwen3-VL 30B-A3B	Class names + instructions	11.9	Table 2
Qwen3-VL 30B-A3B	GEPA	13.0	Table 2
Qwen3-VL 30B-A3B	MIPROv2	10.7	Table 2
Qwen3-VL 30B-A3B	DetPO, self-reported scores	19.4	Table 3
Qwen3-VL 30B-A3B	DetPO + VQA Score	21.6	Table 2
Gemini 3 Pro	Class names + instructions + images	23.8	Table 2
Gemini 3 Pro	GEPA	25.6	Table 2
Gemini 3 Pro	MIPROv2	25.0	Table 2
Gemini 3 Pro	DetPO + Qwen VQA Score	26.3	Table 2

The self-reported-score version improves Qwen3-VL by 7.5 mAP over the 11.9 baseline, with VQA bringing the total improvement to 9.7. Thus, the 9.7 improvement is over the base prompt, not GEPA; relative to GEPA's 13.0, the final version improves by 8.6. The prose and Table 2 caption give Gemini's result as 26.4, whereas the data rows in Tables 2 and 5 give 26.3; this note preserves the data-row value and flags the discrepancy. Likewise, Gemini's image-context baseline is 23.9 in Table 1 but 23.8 in Table 2; these should not be silently unified or used for precise cross-table gain calculations.

Ablation Study

Table 4 on page 10 uses Qwen3-VL 30B-A3B to compare confidence estimators under optimized prompts. Document and Flora & Fauna are domain mAP, while All averages across all datasets.

Config	Document	Flora & Fauna	All
Class names + instructions baseline	7.8	23.5	11.9
Contrastive prompt optimization + self-reported scores	18.6	34.6	19.4
Optimized prompts + SigLIPv2 Score	13.4	28.0	16.4
Optimized prompts + VQA Score	25.2	36.5	21.6

SigLIPv2 scores the cosine similarity between predicted-box crops and class names, reducing All by 3.0 relative to self-reporting; VQA instead improves it by 2.2. This supports instance-specific visual judgment over simple image-text similarity, but the table does not independently isolate prompt-only changes from adding self-reported scores. Table 3 also reports 6.2, 9.1, and 11.9 for the Qwen2.5-VL 7B baseline, DetPO, and VQA settings, showing benefits for a smaller model.

Key Findings

• The TIDE analysis in Figure 6 on page 13 shows reduced class confusion, but localization and missed detections become more prominent bottlenecks after VQA; higher mAP does not mean every error type decreases.
• Table 4 reports medical-domain results of 0.7, 0.1, and 0.2 for the baseline, self-reported-score, and VQA settings, respectively, ruling out a claim of uniform improvement across domains.
• In Table 5 on page 12, fine-tuned GroundingDINO reaches 33.4, above the best black-box prompting result; restricted-access adaptation and weight training are different experimental conditions.
• The main text claims benefits on LVIS, but the numbers are in missing Appendix E, so this note does not repeat unverified quantitative LVIS claims.

Highlights & Insights

• Visual feedback supplies a concrete direction for prompt revision. Unlike a scalar reward alone, it can identify whether confusion arises from background, action relationships, or overly restrictive appearance conditions.
• Using images during adaptation and text during deployment lets a frozen model accumulate task-specific guidance. A useful reader interpretation is compression of annotated experience into inspectable concept descriptions, rather than new knowledge in model weights.
• Definition revision and confidence estimation address different problems. The former affects which objects are detected, while the latter ranks existing predictions, helping separate the sources of improvement.

Limitations & Future Work

• The authors acknowledge substantial prompt-discovery cost, potentially approaching specialist-model training overhead; gradient-free does not mean computation-free, and VQA adds recurring per-box cost.
• When the base model has almost no medical-image capability, better wording cannot easily repair missing visual representations; stronger domain models and prompt improvements should be evaluated separately.
• Incorrectly down-weighting genuine boxes can reduce recall, motivating joint analysis of calibration, localization, and misses instead of optimizing only a single mAP number.
• The authors warn that closed-source APIs may use submitted images and prompts in future training, creating confidentiality and future benchmark-contamination risks; the paper does not establish that such leakage occurred.
• The cache lacks appendices and contains inconsistent Gemini summary values; Table 5's Qwen3-VL 30B-A3B baseline row also disagrees with Tables 2–4, so this note uses the 11.9 baseline from Tables 2–4.

Related Work & Insights

• Versus GEPA and MIPROv2: the general-purpose optimizers in this comparison receive more abstract task feedback, while DetPO supplies boxed visual errors to the critique model; the conclusion concerns this detection setup, not a general ranking of prompt optimizers.
• Versus GroundingDINO and LLMDet: these specialist detectors have mature localization capabilities; DetPO enables concept adaptation for frozen generalist models but has not surpassed a sufficiently fine-tuned specialist.
• Versus multimodal ICL: directly attaching image demonstrations and using demonstrations to search for text prompts are different pathways; Table 1's negative findings concern the former, not an absence of useful information in visual examples.
• Research direction: investigate which false positives and false negatives most effectively change class boundaries, comparing error-sampling strategies at matched call budgets; this is a reader proposal, not a completed ablation in the paper.

Rating

• Novelty: 4/5. Visual comparisons of detection errors provide task-specific discrete prompt revision, while building on existing prompt optimization and VQA scoring.
• Experimental Thoroughness: 4/5. Multiple models, domains, score ablations, and error diagnostics are included, but cost matching and independent component attribution remain limited.
• Writing Quality: 3/5. The main method is clear, but summary values disagree and the current cache lacks appendix implementation details.
• Value: 4/5. A reusable approach for few-shot detection without weight access, with clear limits from localization and base visual capability.