Skip to content

title: >- [Paper Note] Towards Sparsely Annotated Open World Object Detection description: >- [ECCV 2026][Object Detection][Open-World Detection] Introduces SA-OWOD and DPOD to tackle ambiguous supervision from missing known labels and novel unseen categories via KTRM and DDTG. tags: - ECCV 2026 - Object Detection - Open-World Object Detection - Sparsely Annotated Object Detection - Unknown Object Discovery date: 2026-09-19 content_hash: cf2cf60303e05883

Towards Sparsely Annotated Open World Object Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/HelloHeeju/SA-OWOD
Area: Object Detection
Keywords: Open-World Object Detection, Sparsely Annotated Object Detection, Unknown Object Discovery, Pseudo-Label Recovery, Dual-View Semantic Inconsistency

TL;DR

To resolve contradictory supervision arising from ambiguous unlabeled regions in real-world scenarios, this paper introduces Sparsely Annotated Open-World Object Detection (SA-OWOD) and proposes Dual-Perspective Object Discovery (DPOD), which leverages known target recovery and cross-view semantic disagreement to significantly boost unknown object recall while maintaining high known detection accuracy.

Background & Motivation

Practical object detection systems inevitably operate under incomplete and open-ended visual environments. In real-world applications, extensive annotations are hindered by prohibitive costs and human cognitive constraints, yet conventional detectors rest on two restrictive assumptions: training datasets provide complete instance annotations without missing targets, and the target category vocabulary is closed. While Open-World Object Detection (OWOD) attempts to recognize known classes and identify novel unseen instances for incremental learning, existing OWOD techniques presuppose fully annotated data. When deployed under sparse supervision where unlabeled known objects exist, these instances are erroneously treated as background, which distorts decision boundaries and leads the detector to confuse known objects with unknown categories.

Conversely, Sparsely Annotated Object Detection (SAOD) addresses incomplete annotations by recovering unlabeled known objects via pseudo-labeling or self-supervised mining. However, SAOD methods inherently rely on a closed-world assumption: every candidate object is presumed to belong to a predefined set of known categories. Consequently, novel objects outside the target taxonomy are forcibly treated as background or misassigned to known classes, entirely precluding the discovery of new categories. In real-world scenes, unlabeled known instances and truly unknown objects co-exist within the same unlabeled regions and appear visually indistinguishable to the detector.

To address this dilemma, this paper formulates the Sparsely Annotated Open-World Object Detection (SA-OWOD) task, explicitly modeling the competition between missing known instances and emergent unknown categories. Core idea: construct a Dual-Perspective Object Discovery (DPOD) framework that recovers unlabeled known instances through student-teacher pseudo-labeling with Facility-Location feature separation (KTRM), while discovering reliable unknown candidates by detecting cross-view semantic inconsistency via geometrically aligned RoI projections (DDTG), cooperatively disentangling ambiguous supervision.

Method

Overall Architecture

DPOD takes multi-view augmented images with sparse ground-truth annotations as input, built upon a class-agnostic Faster R-CNN backbone with a ResNet-50 feature extractor (derived from CROWD). The pipeline comprises two complementary components: the Known Target Recovery Module (KTRM) refines student predictions via teacher consistency filtering to produce pseudo-labels for unlabeled known instances, while applying Facility-Location conditional gain to separate known and unknown feature embeddings; complementarily, the Dual-Disagreement Target Generator (DDTG) projects region proposals across two augmented views to measure class logit cosine similarity, selecting high-objectness proposals with low semantic agreement as reliable unknown candidates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Sparse Annotations"] --> B["Dual-View Augmentation & Backbone Extraction"]
    B --> C["Class-Agnostic Proposal Generation & Objectness"]
    C --> D["Known Target Recovery Module KTRM<br/>Teacher Consistency Filtering + Pseudo-Labeling"]
    C --> E["Dual-Disagreement Target Generator DDTG<br/>Cross-View RoI Projection & Logit Agreement"]
    D --> F["Known-Unknown Feature Separation<br/>Facility-Location Conditional Gain"]
    E --> G["Unknown Candidate Mining & Disagreement Loss"]
    F --> H["Joint Multi-Task Optimization & Incremental Update"]
    G --> H

Key Designs

1. Known Target Recovery Module (KTRM): Missing Known Instance Recovery and Feature Disentanglement Under sparse supervision, unannotated known objects are mistakenly treated as background, introducing false-negative gradients and blurring class decision boundaries. KTRM leverages a collaborative strong-weak student-teacher paradigm (inspired by Co-Student), filtering student predictions across augmented views through teacher consistency verification to assemble a reliable unlabeled known proposal set \(\mathcal{R}_{uk}\). Each pseudo-label \(p \in \mathcal{R}_{uk}\) contributes a predicted category \(y_p^{uk}\) and bounding box \(\mathbf{b}_p^{uk}\), matched to the highest-IoU proposal \(r_p\) for classification and regression supervision: $\(\mathcal{L}_{uk} = \frac{1}{|\mathcal{R}_{uk}|} \sum_{p \in \mathcal{R}_{uk}} \left[ \ell_{cls}\left(\mathbf{z}_{r_p}, y_p^{uk}\right) + \ell_{reg}\left(\mathbf{b}_{r_p}, \mathbf{b}_p^{uk}\right) \right]\)$ To prevent feature overlap between unlabeled known and unknown proposals in the latent space, KTRM unites sparse annotations \(\mathcal{R}_k\) and pseudo-labels \(\mathcal{R}_{uk}\) into \(\mathcal{R}_{all\_k} = \mathcal{R}_k \cup \mathcal{R}_{uk}\), and enforces a conditional gain separation objective \(\mathcal{L}_{sep}\) based on Facility-Location: $\(\mathcal{L}_{sep} = -\frac{1}{|\mathcal{R}_{all\_k}|} \sum_{i \in \mathcal{R}_{all\_k}} \left( \max_{j \in \mathcal{R}_{all\_k}} s_{ij} - \eta \max_{j \in \mathcal{R}_u} s_{ij} \right)\)$ where \(s_{ij}\) denotes proposal cosine similarity and \(\eta\) controls separation strength. This pushes each known proposal away from its nearest unknown counterpart rather than aggregating over all negative proposals, preventing decision boundary distortion.

2. Dual-Disagreement Target Generator (DDTG): Unknown Object Discovery via Cross-View Semantic Inconsistency In open-world settings, novel categories exhibit significant classification instability across visual perturbations because they reside near ambiguous decision boundaries. Rather than relying on simple IoU matching across views—which is vulnerable to slight localization drifts—DDTG projects RoI proposals from a reference view \(x_{s1}\) geometrically into the paired view \(x_{s2}\). Features extracted from aligned regions produce classification logits \(\mathbf{z}_r^i\) and \(\mathbf{z}_r^j\), whose semantic agreement is evaluated via cosine similarity: $\(\mathcal{S}(r) = \frac{\left(\mathbf{z}_r^i\right)^\top \mathbf{z}_r^j}{\|\mathbf{z}_r^i\|_2 \|\mathbf{z}_r^j\|_2}\)$ Proposals satisfying both high objectness (\(o_r \ge \tau_o\)) and low semantic agreement (\(\mathcal{S}(r) \le \tau_s\)) form the disagreement candidate set \(\mathcal{R}_{dis}\): $\(\mathcal{R}_{dis} = \left\{ r \in \mathcal{R}_a \;\middle|\; o_r \ge \tau_o,\; \mathcal{S}(r) \le \tau_s \right\}\)$ By applying unknown classification loss and conditional feature separation over \(\mathcal{R}_{dis}\), the model actively explores novel category regions without requiring explicit negative class labels.

Loss & Training

The overall training loss for unlabeled region modeling combines KTRM and DDTG objectives: $\(\mathcal{L}_{DPOD} = \gamma_{KTRM} \mathcal{L}_{KTRM} + \gamma_{DDTG} \mathcal{L}_{DDTG}\)$ where \(\mathcal{L}_{KTRM} = \alpha_{uk}\mathcal{L}_{uk} + \alpha_{sep}\mathcal{L}_{sep}\) and \(\mathcal{L}_{DDTG} = \beta_{cls}\mathcal{L}_{cls}^{dis} + \beta_{sep}\mathcal{L}_{sep}^{dis}\). In practice, all weighting hyper-parameters \(\alpha_{uk}, \alpha_{sep}, \beta_{cls}, \beta_{sep}, \gamma_{KTRM}, \gamma_{DDTG}\) and the facility-location factor \(\eta\) are set to 1. The architecture uses a ResNet-50 backbone pre-trained on ImageNet, optimized using AdamW with an initial learning rate of \(2.5 \times 10^{-5}\) and weight decay of \(1 \times 10^{-4}\). Across 4 NVIDIA RTX A6000 GPUs with batch size 12, each task undergoes 15,000 base training iterations followed by 15,000 incremental learning iterations (approx. 9 epochs total).

Key Experimental Results

Main Results

Evaluation follows the standard OWOD benchmark spanning Pascal VOC (Task 1, 20 classes) and MS COCO (Tasks 2–4, 60 classes). Metrics include Known mAP (K-mAP) and Unknown Recall (U-Recall). The table below summarizes performance under full annotation and across representative sparse configurations:

Setting Method Task 1 K-mAP Task 1 U-Recall Task 2 K-mAP Task 2 U-Recall Task 3 K-mAP Task 3 U-Recall Task 4 K-mAP
Full (Upper Bound) CROWD 61.7 57.9 47.8 53.6 42.5 69.6 38.5
Easy RandBox 49.91 4.90 38.36 4.30 33.12 4.60 29.79
Easy PROB 52.83 18.27 38.35 15.35 32.64 19.10 29.06
Easy OrthogonalDet 56.19 17.05 41.91 22.51 36.73 25.13 34.60
Easy CROWD 53.28 45.53 38.24 30.22 35.43 53.34 32.28
Easy DPOD (Ours) 56.48 55.34 39.53 52.47 36.90 63.68 32.15
Hard RandBox 49.96 2.69 27.66 3.62 29.86 3.72 27.26
Hard PROB 50.18 18.26 36.24 13.91 30.98 17.57 27.01
Hard OrthogonalDet 53.14 19.19 38.03 18.28 32.87 24.44 32.14
Hard CROWD 49.29 49.80 34.83 33.35 33.63 50.80 29.91
Hard DPOD (Ours) 54.09 51.85 33.36 48.25 35.79 64.00 31.15
Extreme RandBox 36.62 0.68 21.93 1.50 19.55 2.39 17.95
Extreme PROB 37.78 17.86 26.09 14.83 23.41 17.60 19.38
Extreme OrthogonalDet 41.39 12.41 26.20 14.77 23.30 24.62 22.31
Extreme CROWD 34.37 40.66 24.84 31.17 24.46 48.64 20.47
Extreme DPOD (Ours) 45.37 45.56 25.98 44.32 26.58 57.53 20.19

Ablation Study

Ablation experiments conducted under the Hard configuration on Task 1 analyze the distinct contributions of KTRM and DDTG:

Configuration KTRM DDTG K-mAP (%) \(\Delta\) K-mAP U-Recall (%) \(\Delta\) U-Recall Note
CROWD Baseline ✗ ✗ 49.29 — 49.80 — Baseline detector
+ KTRM ✓ ✗ 51.90 +2.61 50.66 +0.86 Recovers missing knowns; primarily boosts K-mAP
+ DDTG ✗ ✓ 51.08 +1.79 50.26 +0.46 Captures semantic disagreement; refines boundaries
DPOD (Full) ✓ ✓ 54.09 +4.80 51.85 +2.05 Joint dual-perspective modeling achieves best balance

Sensitivity analysis on objectness threshold \(\tau_o\) and similarity threshold \(\tau_s\) in DDTG:

\(\tau_o\) \(\tau_s\) K-mAP (%) U-Recall (%) Findings
0.10 0.95 52.97 47.68 Excess low-objectness background noise deteriorates recall
0.20 0.95 54.09 51.85 Optimal trade-off between proposal purity and candidate coverage
0.30 0.95 53.62 51.40 Overly strict filtering discards subtle novel objects
0.20 0.90 52.24 46.26 Constraining disagreement candidates restricts novel class diversity
0.20 0.85 52.53 50.46 Substantial drop in disagreement sample richness degrades representation

Key Findings

  • Overcoming Unknown Recall Collapse: Prior detectors suffer catastrophic degradation in unknown recall under severe sparsity (e.g., OrthogonalDet falls to 12.41% on Extreme Task 1). DPOD recovers missing known targets via KTRM, eliminating false negatives and reaching 45.56% U-Recall.
  • Synergistic Complementarity: Adding KTRM yields +2.61% K-mAP by neutralizing false background penalties, while DDTG stabilizes decision boundaries around novel objects. Combined, they achieve +4.80% K-mAP and +2.05% U-Recall gains on Hard Task 1.
  • Stage Progression Dynamics: In Task 4, most classes have been absorbed into the known taxonomy, leaving few remaining unknown instances. Consequently, the relative impact of DDTG diminishes, reflecting an expected shift toward known fine-grained classification.

Highlights & Insights

  • First Formulation of SA-OWOD: Unifies the disjoint fields of sparse annotation detection and open-world learning, addressing the realistic setting where missing labels and novel classes co-occur.
  • Geometric Cross-View RoI Projection: Replaces noisy bounding box IoU heuristics with exact spatial coordinate projection across augmented views, ensuring semantic inconsistency reflects true category ambiguity rather than localization jitter.
  • Facility-Location Feature Separation: Applies nearest-prototype negative repulsion instead of broad global contrastive push, preserving compact within-class manifolds under sparse supervisory signals.

Limitations & Future Work

  • Static Heuristic Thresholds: DDTG currently employs fixed objectness (\(\tau_o=0.2\)) and similarity (\(\tau_s=0.95\)) thresholds. For categories with low intrinsic objectness scores (e.g., thin accessories like neckties), fixed thresholds risk systematic omission.
  • Backbone Scope: Evaluations rely exclusively on Faster R-CNN with ResNet-50; expanding to open-vocabulary vision-language models (e.g., OWL-ViT) or query-based DETR architectures remains unexplored.
  • Future Directions: Developing learnable, class-adaptive thresholds for disagreement filtering and extending dual-view semantic inconsistency to dense query decoders.
  • vs CROWD (Majee et al., NeurIPS 2025): CROWD separates known and unknown categories using submodular facility location, but presumes complete annotations. When missing annotations occur, KTRM provides essential pseudo-label recovery to prevent unlabeled knowns from polluting the unknown pool.
  • vs OrthogonalDet (Sun et al., CVPR 2024): OrthogonalDet enforces orthogonality between objectness and classification features, but its unknown recall drops by over 43% under extreme sparsity. DPOD maintains robust unknown discovery via cross-view disagreement modeling.
  • vs Co-Student (Wu et al., ECCV 2024): Co-Student mines missing annotations within closed-world datasets. DPOD adapts this concept for open-world settings, combining it with perturbation instability metrics to enable novel class discovery.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the realistic SA-OWOD problem and introduces an elegant cross-view semantic disagreement mechanism.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across five sparsity configurations spanning Easy to Extreme settings with comprehensive ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, cohesive structure, and disciplined experimental analysis.
  • Value: ⭐⭐⭐⭐⭐ Bridges a critical gap between academic assumptions and real-world deployment for open-world perceptual systems.