MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection¶

Conference: ECCV 2026
arXiv: 2510.14792
Code: None
Area: Object Detection
Keywords: Open-Vocabulary Object Detection, Pseudo-Labeling, Multi-Step Reasoning, Contrastive Learning, MLLM

TL;DR¶

MSPL reformulates pseudo-label generation for open-vocabulary object detection from single-step CLIP image-text alignment into an decodable, three-step reasoning pipeline: "localization verification -> category identification -> background grounding". Utilizing SAM category-agnostic segmentation and MLLM zero-shot inference, it generates high-quality pseudo-labels. These are then used during online training via two contrastive learning signals: region-text alignment and contrastive background learning. It achieves new SOTA performance on both OV-COCO and OV-LVIS, with a 9.4-point gain in novel-class AP50.

Background & Motivation¶

Open-vocabulary object detection (OVD) aims to simultaneously localize both seen base and unseen novel objects at test time, while only base-class annotations are available during training. To bridge this supervision gap, pseudo-labeling has recently emerged as a mainstream approach—leveraging vision-language models (VLMs) like CLIP to perform single-step similarity matching between novel-class regions and text embeddings, automatically generating pseudo-annotations to expand the training set. While these methods perform well in simple scenarios, they degrade rapidly in complex scenarios involving crowding and occlusion.

In-depth analysis reveals three root-cause failure modes. The first is co-occurrence interference: single-step alignment relies on VLMs trained under image-level supervision, where regional features naturally encode surrounding contextual statistics; consequently, a partially occluded foot might be mislabeled as a "skateboard" simply because the skateboard strongly co-occurs in the image. The second is caption dependency: single-step matching requires a pre-defined candidate category set, typically extracted from image captions; any object not appearing in the captions (as well as those submerged in coarse-grained descriptions) is by design impossible to detect. The third is background collapse: identifying occluded objects requires first identifying the occluders and then inferring the occluded objects; single-step alignment circumvents this decomposition, causing unassigned object regions to be directly learned as background. These three issues share a common root—single-step alignment compresses the entire scene into a single reasoning unit, leaving no room to decouple co-occurrences, discover unspecified categories, or reason about occlusions.

The core insight of this paper is: since complex visual understanding naturally demands multi-step reasoning, why can't pseudo-label generation also proceed step-by-step? The authors propose to reformulate pseudo-label generation into an interpretable three-step visual prompting process—first validating whether a region indeed contains an object, then performing zero-shot category assignment and generating natural language descriptions, and finally explicitly determining whether the region belongs to the foreground or background—where the outputs of each step serve as rich supervisory signals for subsequent online training. Core Idea: Replace single-step CLIP alignment with a three-step progressive reasoning process ("localization verification -> category identification -> background grounding") to generate OVD pseudo-labels. Each engineered step yields not only the final labels but also intermediate reasoning states (regional descriptions, background determinations) as supervision for contrastive learning, achieving high-quality pseudo-annotations in highly occluded and crowded scenarios.

Method¶

MSPL is a framework featuring offline pseudo-label generation and online contrastive learning. In the offline phase, SAM is used to generate category-agnostic region proposals, which are fed into an MLLM to perform three-step reasoning after visual context modulation; finally, semantic anchors are filtered using a frequency threshold. In the online phase, these pseudo-labels and their intermediate states are formulated into contrastive learning losses to train a Faster R-CNN detector. Offline multi-step reasoning shifts the heavy visual computation overhead to the pre-training phase, leaving only lightweight contrastive learning online, which eliminates the overhead of iterative online self-training.

Overall Architecture¶

The entire framework comprises two phases: offline pseudo-label generation and online training. In the offline phase, given training images, candidate regions are segmented via SAM, followed by a three-step MLLM reasoning process to sequentially perform object verification, zero-shot label assignment, and background determination. Finally, the semantic anchor set is filtered based on base-class statistical frequencies. In the online phase, semantic anchors serve as positive samples for region-text alignment (RTA) alongside the region descriptions generated by the MLLM in the second step. Concurrently, background concepts identified in the third step act as negative samples for contrastive background learning (CBL). Additionally, combinatorial augmentation is achieved by caching anchors to replace the online neighbor sampling of the baseline BARON, accelerating training. During inference, pseudo-labels are discarded, and classification is performed utilizing only base classes combined with CLIP text embeddings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Training Images"] --> B["SAM Category-Agnostic Segmentation<br/>Hierarchical Grouping -> Whole Instance"]
    B --> C["Visual Context Modulation<br/>Background Blurring + Grayscale"]
    C --> D["Step 1: Object Localization Verification<br/>MLLM Tri-value Determination"]
    D -->|"Yes"| E["Step 2: Category Identification<br/>MLLM Outputs Label + Description"]
    D -->|"No/Unsure"| F["Discard"]
    E --> G["Step 3: Background Grounding<br/>Foreground/Background Binary Classification"]
    G --> H["Semantic Anchor Filtering<br/>Filter by Base Class Min Frequency Threshold"]
    H --> I["Semantic Anchor Set<br/>Label + Description + Background Det."]
    I --> J["Online Contrastive Learning Training"]
    J --> K["ℒ_RTA<br/>Region-Text Alignment"]
    J --> L["ℒ_CBL<br/>Contrastive Background Learning"]
    J --> M["Combinatorial Augment<br/>Cache bag-of-regions"]
    K --> N["Faster R-CNN Detector"]
    L --> N
    M --> N

Key Designs¶

1. Progressive Three-Step Reasoning: Decomposing Pseudo-Label Generation into Explainable Step-by-Step Cognition

The fundamental flaw of single-step CLIP alignment lies in compressing scene understanding into a single match. MSPL decomposes this process into three independent cognitive steps, each with a clear reasoning objective and intermediate outputs. In the first step (localization verification), upon receiving SAM's category-agnostic mask, the system does not directly classify it semantically. Instead, it queries the MLLM: "Does this region contain an object?" and receives a tri-value determination (Yes/No/Unsure). This hard gating mechanism ensures that subsequent steps only perform semantic reasoning on regions that truly contain objects, preventing semi-instance fragments generated by SAM (e.g., local parts, shadows) from being mislabeled. The second step (category identification) discards the candidate vocabulary sets relied upon by traditional methods (usually extracted from image captions). It prompts the MLLM to predict category names in a zero-shot manner and generate a natural language description (e.g., "a brown and white dog with long, wavy ears sitting"), making it possible to discover any object present in the image without preset vocabulary constraints. The third step (background grounding) performs a secondary check on predictions that lacked high confidence in the preceding steps. It uses the MLLM to judge whether the prediction is a foreground object or a background concept (e.g., "grass", "sky"), explicitly preventing unannotated objects from being absorbed into the background embedding. A clear cognitive gradient is formed across these three steps: from "Is there something?" to "What is it?" and finally "Is it foreground or background?" The output of each step (tri-value determination, label + description, foreground/background tag) serves as additional supervision for online training.

2. Region-Text Alignment (RTA): Enhancing Contrastive Learning Signals with Descriptive Language

Relying solely on category names for contrastive learning easily leads to confusion among fine-grained visually similar categories (e.g., a red "apple" vs. a red "ball"). The region descriptions generated by the MLLM in the second step compensate for this granularity limitation: descriptions encompass not only category names but also attributes such as color, texture, pose, and part relations. RTA aligns the pseudo-labeled region features (pseudo-word embeddings) with their corresponding descriptive text embeddings via InfoNCE contrastive alignment. Specifically, for candidate regions with IoU > 0.7, within the same batch, the current region's pseudo-word embedding forms a positive pair with its description embedding and negative pairs with the description embeddings of other regions:

\[\mathcal{L}_{\text{RTA}} = -\frac{1}{|\mathcal{S}|} \sum_{k \in \mathcal{S}} \log \frac{\exp(\tau \cdot \langle f_t^k, f_d^k \rangle)}{\sum_{l \in \mathcal{S}} \exp(\tau \cdot \langle f_t^k, f_d^l \rangle)}\]

Where \(f_t^k\) is the pseudo-word text embedding of the \(k\)-th region, \(f_d^k\) is its corresponding description embedding, and \(\tau\) is the temperature coefficient. This loss directly encourages regional visual features to align closely with their multi-dimensional attribute descriptions in the CLIP embedding space, compelling the model to learn richer discriminative dimensions rather than simply memorizing a category name.

3. Contrastive Background Learning (CBL): Explicitly Decoupling Foreground and Background Embeddings

The essence of background collapse is that unannotated object regions are implicitly pushed toward the background representation in the feature space. Traditional methods model the background as a unified "disinterested" embedding, which swallows up occluded objects. MSPL addresses this by exploiting the background concepts identified in the third step (e.g., sky, water surface, vegetation, paved ground, plain wall): by averaging the CLIP text embeddings of these background concepts, it initializes a learnable background prior \(f_{bg}\). Then, during training, it requires the text-visual alignment of each bag-of-regions to not only pull positive pairs closer and push other negative pairs apart but also push away the background embedding. This establishes a structured, identifiable feature cluster for the "background," rather than a generic bin that absorbs all non-targets. Based on bidirectional InfoNCE, the CBL loss explicitly introduces background negative samples into the contrastive pool, forcing foreground object features away from the background cluster, thereby recovering object features that would otherwise be obscured by the background. Empirical results demonstrate that CBL effectively resolves the issue of targets being swallowed by the background in occluded scenarios.

Loss & Training¶

The total loss in the online phase consists of three parts: the original Faster R-CNN regression and classification loss, the RTA (region-text alignment) loss, and the CBL (contrastive background learning) loss. The backbone network is a SOCO-pretrained ResNet-50 FPN (ResNet-50×4 FPN is used for large-model experiments), utilizing synchronized batch normalization. OV-COCO uses a 1× training schedule (90k iterations), while OV-LVIS uses a 2× training schedule (180k iterations), with learning rates of 0.04 and 0.08 respectively, and a batch size of 16. The three temperature parameters are set to \(\tau=0.2\), \(\tau'=0.05\), and \(\tau''=0.1\). For combinatorial augmentation, MSPL groups regions into cached bag-of-regions using semantic anchors, replacing the online neighbor sampling of BARON, which accelerates training speed by 1.5×. During inference, pseudo-labels are discarded, and classification is performed utilizing only base classes combined with CLIP text embeddings.

Key Experimental Results¶

Main Results¶

On OV-COCO, MSPL achieves 43.4 AP50_N using a ResNet-50 backbone and pseudo-annotation supervision, significantly outperforming concurrent pseudo-labeling methods (LP-OVOD 40.5, SAS-Det 37.4). When the backbone is scaled up to ResNet-50×4, it reaches 47.8 AP50_N, surpassing the previous state-of-the-art method OV-DQUO (45.6).

Method	Backbone	Novel AP50_N	Base AP50_B
ViLD-ens	RN50	27.6	51.3
BARON	RN50	34.0	60.4
LP-OVOD	RN50	40.5	60.5
MSPL	RN50	43.4	58.9
CLIP-Self	ViT-L/14	44.3	-
MSPL	RN50×4	47.8	60.9

On OV-LVIS (featuring a long tail of 337 novel classes), MSPL achieves a detection APr of 26.4 and a segmentation APr of 24.8, outperforming the previous best CAKE by 1.4 and 0.9 points, respectively.

Method	Detection APr	Detection AP	Segmentation APr	Segmentation AP
BARON	23.2	29.5	22.6	27.6
LBP	24.1	29.9	23.7	28.0
CAKE	25.0	34.9	23.9	28.7
MSPL	26.4	34.9	24.8	28.6

In terms of zero-shot cross-dataset transfer, MSPL trained on OV-LVIS and evaluated directly on MS-COCO and Objects365 also comprehensively outperforms previous methods.

Ablation Study¶

Configuration	OV-COCO AP50_N	Description
Baseline (BARON)	34.0	No pseudo-labels
+ 1-Step PL	37.6	SAM+MLLM single-step reasoning
+ 3-Step PL	41.6	Three-step reasoning pipeline
+ RTA	42.5	Add region-text alignment
+ CBL	43.4	Add contrastive background learning

MLLM Comparison	Parameters	AP50_N
BLIP2	2.7B	39.6
InstructBLIP	7B	42.6
Qwen2-VL	7B	43.4

Contextualization Strategy	AP50_N
Bounding box only	33.2
Black mask (all black outside region)	38.7
Background blur + grayscale	43.4

Key Findings¶

Three-step reasoning contributes the most: Three-step reasoning (41.6) yields a 4.0-point improvement over single-step reasoning (37.6), which is significantly larger than RTA (+0.9) and CBL (+0.9). This indicates that decomposed cognition is the key contributor to understanding complex scenes.
Visual context modulation is crucial: Doing nothing (bounding box only) yields only 33.2, whereas background blurring + grayscale outperforms direct black-out masking by nearly 5 points. This suggests that preserving a moderate amount of context assists MLLM reasoning, while complete context deprivation triggers hallucinations.
Positive correlation between MLLM capability and performance: Upgrading from the 2.7B BLIP2 to the 7B Qwen2-VL increases AP50_N from 39.6 to 43.4, with negligible differences between architectures at equivalent scales.
MIN frequency threshold is optimal: Filtering with the minimum annotation frequency of base classes (rather than the mean or median) achieves the best performance. This reveals that long-tail, low-frequency pseudo-labels suffer from severe noise and require strict gating.

Highlights & Insights¶

"Cognitive Decomposition" over "Model Stacking": The most inspiring aspect of MSPL is that it is not a simplistic combination of "SAM candidate extraction + MLLM labeling". Instead, it interpolates a meticulously designed structure between them—comprising three-step reasoning, tri-value validation, and visual context modulation—allowing the two foundation models to collaborate rather than interfere. This reasoning-structure-centric blueprint is more worthy of emulation than merely scaling up model size.
Intermediate States as Supervision: The tri-value determination signals, "unsure" tags, and natural language descriptions generated by the MLLM are all utilized as auxiliary supervision signals, rather than being discarded. This implies that every "byproduct" of the pseudo-labeling pipeline holds potential teaching value.
Decomposed Modeling of Background Concepts: CBL's main insight is decomposing the "background" from a single embedding into five semantically defined categories (sky, water surface, vegetation, paved ground, plain wall) to serve as negative samples. This injects structure into the background representation rather than treating it as an undifferentiated garbage bin.

Limitations & Future Work¶

Inherent Dependency on MLLM Capabilities: MSPL's performance is intrinsically limited by the reasoning quality of the underlying MLLM. Although the authors recommend at least a 7B parameter size, the model remains unstable in recognizing long-tail, ultra-fine-grained objects and abstract concepts. For example, MLLMs tend to over-rely on shape while ignoring color (classifying all knife-shaped objects as knives), and struggle to generalize proper nouns correctly (e.g., "Eiffel Tower" vs. "building").
Wastage of "Unsure" Responses: The current strategy completely discards "Unsure" responses to guarantee precision, but these responses may convey valuable long-tail category signals. Future work could explore incorporating low-confidence predictions into training via soft labels.
Single-Frame Limitation: Currently, the framework only operates on static images. However, resolving occlusions and tracking object persistence naturally require temporal information. The authors point toward adding a fourth step of temporal reasoning to extend the model to video detection in the future.

vs VL-PLM / SAS-Det / PB-OVD: These methods all rely on single-step CLIP matching to generate pseudo-labels, depending on image captions to define candidate vocabularies. In contrast, MSPL utilizes MLLM for zero-shot assignment, freeing it from caption dependence and enabling it to handle occlusion and crowding. This yields a leading margin of 10+ AP50_N points in complex scenarios.
vs BARON: Implementing BARON as its baseline, MSPL replaces BARON's online neighbor sampling with cached semantic anchors, accelerating training speed by 1.5×. More importantly, it adds an offline pseudo-label generation pipeline, substantially outperforming BARON under heavy occlusion.
vs CLIP-Self / DeCo-DETR: These are distillation/self-training methods that solely exploit CLIP knowledge and base-class annotations. MSPL secures a 2.1-6.2 AP50_N margin of victory through auxiliary pseudo-annotations, demonstrating the value of the pseudo-labeling paradigm, albeit at the expense of computational costs in annotation generation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Introducing multi-step visual reasoning into OVD pseudo-label generation, using a three-step decomposition and background grounding design that is intuitive and effective. However, individual components (SAM, MLLM, contrastive learning) are already mature tools, meaning the innovation lies primarily in structural integration and the utilization of intermediate states.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Demonstrates SOTA performance in main experiments across two benchmarks (OV-COCO, OV-LVIS), complemented by comprehensive ablations on zero-shot cross-dataset transfer, MLLM variants, contextualization strategies, and semantic anchor strategies.
Writing Quality: ⭐⭐⭐⭐☆ The exposition is clear, particularly the intuitive and powerful visualization of the three failure modes in Figure 3. However, the explanations of background collapse and the formulation of the RTA loss function in the main text could be more direct.
Value: ⭐⭐⭐⭐⭐ While the OVD pseudo-labeling domain has long been dominated by single-step approaches, MSPL demonstrates the promise of trading reasoning complexity for annotation quality. This exerts a direct positive impact on detecting objects under heavy occlusion and heavy crowding in industrial scenarios.