Real-Time Source-Free Object Detection

Conference: ECCV 2026
arXiv: 2606.31834
Code: https://github.com/Sairam13001/RT-SFOD/
Area: Autonomous Driving / Source-Free Domain Adaptive Object Detection
Keywords: Source-Free Domain Adaptation, Real-Time Object Detection, YOLO, Dual-Head Detector, Pseudo-Label Fusion

TL;DR

This paper constructs RT-SFOD, a real-time source-free domain adaptive object detection framework based on YOLOv10 (the first NMS-free dual-head detector). It proposes Dual-Head Pseudo-Label Fusion (DHF) to recover objects missed by the O2O head while maintaining high precision, alongside a Multi-Scale Adaptive Representation Diversification Loss (MARD) to combat feature rank degradation caused by domain shift. On four domain shift benchmarks, it achieves a 1.4-3.5% mAP improvement with 1.3x faster inference speed and ~2x parameter compression.

Background & Motivation

Background: In practical applications such as autonomous driving and surveillance, Source-Free Object Detection (SFOD) requires adapting a pre-trained detector to a target domain without accessing the source data due to privacy constraints. Most existing SFOD methods are built on heavy backbones like Faster R-CNN or DETR. While these methods continuously improve adaptation accuracy, they completely overlook inference speed (FPS) and model size, failing to meet the strict latency and memory constraints of real-time applications.

Limitations of Prior Work: A natural direction for improvement is migrating SFOD to one-stage lightweight detectors like the YOLO series. However, the state-of-the-art YOLOv10 introduces an NMS-free dual-head design—using an O2O (one-to-one matching) head for end-to-end inference without post-processing, and an O2M (one-to-many matching) head to provide stronger training supervision. When directly applying the standard Mean-Teacher (MT) self-training framework, two issues arise: (1) The pseudo-labels from the O2O head have extremely high precision (0.923) but low recall (0.446), missing many objects; the O2M head has a slightly higher recall (0.469) but introduces extra noise (precision 0.847). Using either head alone or simply merging their predictions yields suboptimal results. (2) Domain shift causes a significant drop in the effective rank of multi-scale feature maps. The authors define effective rank using information entropy and verify this drop, showing that the representational diversity of features is severely compressed after domain shift. Standard MT self-training can only recover less than half of this rank loss, which forms an upper bound on adaptation performance.

Key Challenge: The precision-recall complementarity in dual-head detectors and the degradation of feature discriminative power under domain shift are two bottlenecks that existing MT frameworks cannot handle simultaneously. While O2O achieves high precision but low recall, and O2M improves recall at the cost of noise, at the feature level, domain shift collapses the originally highly discriminative feature space into a low-dimensional subspace. The EMA smoothing in MT is insufficient to recover this structural degradation.

Goal: Maintaining the NMS-free inference efficiency of YOLOv10 (without adding any inference overhead) while addressing the aforementioned two training-stage bottlenecks to elevate the Pareto frontier of accuracy, speed, and model size.

Key Insight: The authors observe that the dual-head pseudo-label issue is essentially about "how to use the precise predictions of one head as anchors to selectively absorb new objects supplemented by the other head" instead of a naive merging. Meanwhile, the feature degradation problem can be counteracted by imposing structured variance-covariance constraints on the multi-scale PAN features within the detection architecture. This is similar to the collapse-prevention ideas in self-supervised learning (e.g., VICReg) but needs to be adapted to detection-specific foreground/background sampling and scale allocation.

Core Idea: Using a fusion strategy of "O2O anchors + non-redundant O2M supplements" to enhance pseudo-label quality, while imposing detection-aware variance-covariance regularization on PAN detection features to recover feature discriminability.

Method

Overall Architecture

RT-SFOD embeds two training-time-only modules (DHF and MARD) into the Mean-Teacher framework. During inference, it completely reverts to the standard YOLOv10 without any extra overhead. The adaptation pipeline is as follows: first, AdaBN (Adaptive Batch Normalization) is applied on the target domain images to update the BN statistics, initializing both the teacher and student models. Then, the model enters an epoch-by-epoch self-training loop. The teacher generates O2O and O2M predictions on weakly augmented views, which are fused via DHF to generate pseudo-labels. These pseudo-labels are mapped to strongly augmented views to supervise the student. During the student forward pass, the MARD loss is additionally calculated on three PAN feature map levels. The total loss consists of the standard YOLOv10 detection loss and the adaptively weighted MARD loss. At the end of each epoch, the teacher is updated once with an EMA momentum of 0.999 (instead of step-by-step, as the authors find that updating once per epoch performs best).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Domain Images"] --> B["AdaBN Warm-up<br/>Initialize Teacher and Student"]
    B --> C["Weakly Augmented View<br/>Teacher Forward"]
    C --> D["Dual-Head Pseudo-Label Fusion (DHF)<br/>O2O Anchors + Non-redundant O2M"]
    D --> E["Map to Strongly Augmented View<br/>as Pseudo-Label Supervision"]
    E --> F["Strongly Augmented View<br/>Student Forward"]
    F --> G["Detection Loss<br/>O2O + O2M Dual-Head"]
    F --> H["Multi-scale Adaptive<br/>Representation Diversification Loss"]
    G --> I["Total Loss = Detection Loss + λ · MARD"]
    H --> I
    I --> J["Update Student Parameters"]
    J --> K["At End of Each Epoch<br/>EMA Update Teacher"]
    K -->|Next Epoch| C
    K --> L["Inference: Standard YOLOv10<br/>O2O Head Only, NMS-free"]

Key Designs

1. DHF (Dual-Head Pseudo-Label Fusion): Selective absorption of non-redundant O2M detections using O2O high-precision anchors as priors

The O2O head generates non-redundant, high-precision predictions but suffers from severe missed detections, while the O2M head offers wider coverage but is highly noisy. Simply merging them achieves suboptimal results in both precision and recall (F1 score of only 0.634, and direct fusion even performs worse in mAP than O2O alone). The core idea of DHF is to treat O2O predictions as immutable "anchors" and only allow O2M to fill in the areas not covered by them.

Specifically, the process consists of three steps: (1) O2O predictions are confidence-filtered (\(s(p) \geq \tau_o\), default 0.5) to serve as a high-quality anchor set \(\hat{Y}^o\). (2) Candidates with sufficient confidence (\(s(p) \geq \tau_m\)) are filtered from O2M, but only those whose IoU with any O2O anchor does not exceed \(\tau_{no}\) (default 0.2) are retained, representing "new objects that do not overlap with any anchors". (3) Since O2M candidates might still contain redundant duplicates, class-wise NMS (threshold \(\tau_{dup}=0.7\)) is applied to these extra candidates for deduplication. The final pseudo-label set is the union of the O2O anchors and the deduplicated O2M supplementary boxes.

The ingenuity of this strategy lies in that O2O acts as the primary "prior", guaranteeing that no O2O box is replaced or contaminated by O2M noise. Meanwhile, O2M only plays a "gap-filling" role, and its intrinsic noise is significantly suppressed through the dual filtering of the IoU gate and NMS. Consequently, DHF pseudo-labels maintain a precision of 0.919 (close to the 0.923 of pure O2O) while pushing the recall from 0.446 to 0.525, achieving an F1 score of 0.668 (a 10.7% relative improvement).

2. MARD (Multi-Scale Adaptive Representation Diversification): Foreground/background-aware variance-covariance regularization on PAN detection features

Domain shift significantly reduces the effective rank of YOLOv10's three PAN feature map levels (P3/P4/P5), indicating that the effective information dimension of feature vectors is compressed. This results in high inter-channel redundancy, shrunk single-channel variance, and less discriminative information available for the detection heads. Standard MT self-training provides limited recovery (reclaiming only 26-48% of the rank loss).

MARD's solution is to sample foreground and background feature vectors from each PAN feature map during the student's forward pass, and then apply two complementary constraints: (1) The variance term \(\mathcal{L}_{var} = \frac{1}{C}\sum_{c=1}^{C}\max(0, \gamma - \sqrt{\mathrm{Var}(Z_{:,c}) + \epsilon})\), which forces each channel to maintain a standard deviation of at least \(\gamma\) to prevent feature collapse. (2) The covariance term \(\mathcal{L}_{cov} = \frac{1}{C(C-1)}\sum_{i\neq j}(\mathrm{Cov}(\tilde{Z})_{ij})^2\), which penalizes correlation between different channels to reduce redundancy.

Unlike collapse-prevention methods in general representation learning such as VICReg, MARD introduces three key adaptations: (1) Sampling is not global; it employs \(K\) spatial points inside the pseudo-labeled boxes as foreground and \(M\) points outside the boxes as background, directly applying regularization to the features leveraged by the detection task. (2) Foreground sampling only selects the top-\(K_b\) (default 15) pseudo-boxes sorted by confidence to mitigate the contamination of noisy pseudo-labels on regularization. (3) Each pseudo-box is assigned to its corresponding PAN scale (e.g., small objects to P3, large objects to P5) based on its pixel area \(s(b)=\sqrt{w \cdot h}\) and stride thresholds, avoiding cross-scale mixed sampling that disrupts scale specificity.

The MARD loss \(\mathcal{L}_{mard} = \sum_{\ell \in \{3,4,5\}}(\alpha\mathcal{L}_{var}(Z_\ell) + \beta\mathcal{L}_{cov}(Z_\ell))\) is also scaled by an adaptive weight \(\lambda(t) = \lambda_0 \cdot \mathrm{ramp}(t) \cdot \mathrm{gate}(\bar{s})\): where \(\mathrm{ramp}(t)\) linearly warms up during the early training phase (first 5 epochs), and \(\mathrm{gate}(\bar{s})\) dynamically scales the intensity based on the batch-averaged pseudo-label confidence. If pseudo-labels are of poor quality, MARD automatically scales down to prevent regularizing on noise. Additionally, MARD is computed every \(I\) steps to control computational overhead.

Loss & Training

The total loss is \(\mathcal{L} = \mathcal{L}_{det} + \lambda(t) \cdot \mathcal{L}_{mard}\), where \(\mathcal{L}_{det}\) represents the standard YOLOv10 detection loss (bounding box regression + classification + distribution focal loss), computed simultaneously on both the student's O2O and O2M heads to sustain dual-head training dynamics. The student model is optimized using SGD with an initial learning rate of 1e-4, employing cosine annealing, a gradient clipping norm of 10, a batch size of 16, and is trained for 60 epochs. The teacher model uses an EMA momentum of 0.999 and is updated once per epoch (instead of every step). All hyperparameters are kept consistent across all four domain shift benchmarks, eliminating the need for scene-specific parameter tuning.

Key Experimental Results

Main Results

Cityscapes \(\rightarrow\) Foggy Cityscapes (C2F, weather domain shift) is the most critical benchmark for SFOD. The table below excerpts representative methods alongside three scale variants of the proposed method.

Method	Backbone	Params (M)	FPS	mAP	Description
Source-only (YOLOv10S)	YOLOv10S	7.2	233	29.6	No adaptation baseline
Source-only (YOLOv10L)	YOLOv10L	24.4	67	36.5	Large model is still inferior to all SFOD methods
IRG (CVPR'23)	FRCNN	34.0	51	37.1	Early SFOD benchmark
Simple-SFOD (ECCV'24)	FRCNN	43.8	42	45.0	One of the best among Faster R-CNN methods
FALCON-SFOD (CVPR'26)	FRCNN	43.8	42	46.9	Introduces foundation model priors
SF-YOLO-L (ECCV'24 ws)	YOLOv5L	46.5	52	51.6	Previous state-of-the-art SFOD accuracy
DRU (ECCV'24)	Def-DETR	40.0	29	43.6	DETR-based SFOD
FRANCK (TIP'25)	Def-DETR	40.0	29	44.9	DETR-based SFOD
VFM-SFOD (AAAI'26)	Def-DETR	41.0	27	47.1	DETR-based SFOD
RT-SFOD-S (Ours)	YOLOv10S	7.2	233	44.3	Smallest model, fastest
RT-SFOD-M (Ours)	YOLOv10M	15.4	105	47.4	Comparable accuracy to VFM-SFOD, 3.6x faster
RT-SFOD-L (Ours)	YOLOv10L	24.4	67	53.8	New SOTA, outperforms SF-YOLO-L by 2.2 points

Core results on the other three domain shift benchmarks: On Sim10k \(\rightarrow\) Cityscapes (S2C, synthetic-to-real), RT-SFOD-L reaches 71.2 AP, surpassing SF-YOLO-L (69.8). On KITTI \(\rightarrow\) Cityscapes (K2C, camera shift), RT-SFOD-L achieves 60.9 AP, second only to SF-YOLO-L (63.7) which has 1.9x more parameters. On Cityscapes \(\rightarrow\) BDD100k (C2B, large-scale diverse shift), RT-SFOD-L reaches 46.5 mAP, outperforming VFM-SFOD (43.0) by +3.5.

Ablation Study

The table below shows the component ablation of RT-SFOD-M across three benchmarks (applied progressively).

Configuration	C2F mAP	S2C AP	K2C AP	Description
Source-trained	30.5	52.5	40.3	No adaptation baseline
+ AdaBN Warm-up	37.4	54.2	45.2	BN statistics alignment, providing better initialization
+ MT (O2O) Self-training	45.2	62.1	50.6	MT baseline using O2O pseudo-labels only
+ DHF	46.7 (+1.5)	65.1 (+3.0)	53.0 (+2.4)	Dual-head fusion improves pseudo-label quality
+ MARD	46.3 (+1.1)	65.3 (+3.2)	52.5 (+1.9)	MARD alone, without DHF assistance
+ DHF + MARD (Full)	47.4 (+2.2)	66.4 (+4.3)	54.2 (+3.6)	The two are complementary, cumulative gains exceed the sum of individual gains

Ablation of MARD sub-components (with MT(O2O)+DHF as baseline): The variance term alone yields the largest contribution (+0.5 on C2F, +0.8 on S2C, +0.8 on K2C), consistent with the hypothesis that channel variance collapse is the primary mode of rank degradation. The covariance term provides a complementary contribution (+0.3/+0.6/+0.5). Their combination matches the full effect of MARD (+0.7/+1.3/+1.2).

Key Findings

• DHF and MARD address complementary bottlenecks: DHF improves the quality of supervision signals, while MARD enhances the quality of feature representation. Their combined gain (+4.3 on S2C) exceeds the sum of individual gains, verifying their complementarity.
• In the pseudo-label strategy comparison, directly merging O2O with O2M+NMS actually degrades mAP (F1 0.634, mAP 45.5 vs. 45.2 for O2O-only), indicating that noisy pseudo-labels accumulate and amplify during iterative self-training—an issue effectively averted by DHF via IoU gating.
• Extremely low hyperparameter sensitivity: The four key hyperparameters (\(\tau_o\), \(\tau_m\), \(\lambda_0\), \(\mu\)) cause fluctuations of only \(\leq 1.5\) mAP within their test ranges. The entire hyperparameter set is directly reused across all four benchmarks and three model scales without scene-specific tuning.
• Generalization verification: RT-SFOD consistently improves the MT(O2O) baseline by 2.0-2.8 mAP on YOLOv26S/M/L, MS-DETR, and Mr. DETR, proving that DHF and MARD serve as general solutions for the dual-head detector paradigm, rather than being YOLOv10-specific.

Highlights & Insights

• "O2O Anchors + O2M Gap-filling" Fusion Strategy: Instead of a naive confidence weighting or simple union, it treats O2O as an inviolable prior via IoU gating, letting O2M function solely in blank regions. This design is clean, explainable, introduces no extra learnable parameters, and generalizes across architectures over multiple dual-head detectors. This paradigm is transferable to any multi-head model with complementary prediction branches (e.g., multi-scale prediction fusion, multi-expert model ensembles).
• Quantifying Domain Shift Impact with Effective Rank: Introducing the concept of "effective rank" from information theory (exponential entropy based on singular value spectrum) to the feature analysis of detection tasks. This quantitatively reveals how domain shift systematically compresses the feature space. This analytical method can serve as a general diagnostic tool for understanding "what exactly went wrong" during domain adaptation.
• Detection-Aware Regularization Instead of Global Regularization: Unlike VICReg or Barlow Twins that apply regularization on global embeddings, MARD employs pseudo-boxes to guide foreground/background sampling and assigns them to corresponding PAN layers based on box size. This ensures the regularization precisely targets "the exact features utilized by the detection heads." The philosophy of "adapting general representation learning techniques to the internal structure of detection" can be transferred to other dense prediction tasks (e.g., segmentation, pose estimation).
• Training-Time-Only Overhead, Zero Inference Cost: Both DHF pseudo-label fusion and MARD feature regularization function as auxiliary supervisory signals during training. At inference, the model completely reverts to the standard YOLOv10 without architectural modifications, extra branches, or modifications to the inference pipeline. This is highly deployment-friendly, and the authors quantitatively evaluated the training overhead (+17.1% time per epoch, +1470MB GPU memory), which is marginal compared to the accuracy gains.

Limitations & Future Work

• Applicable Only to Dual-Head Detectors: DHF relies on the existence of two complementary O2O and O2M heads. For traditional single-head YOLOs (such as YOLOv5/v8/v11) or single-head DETR series, DHF degrades to simple confidence-threshold-based pseudo-label filtering, losing the fusion gain. However, MARD remains applicable.
• Failure Cases under Extreme Scenarios: In extreme weather conditions combining heavy fog and solar overexposure, or under severe occlusion of dense small objects, RT-SFOD still suffers from missed detections or imprecise localization. However, these extreme cases inherently present highly weak visual cues and reflect intrinsic tasks difficulties rather than methodological flaws.
• Camera Domain Shift Reliance on Model Capacity: On K2C, RT-SFOD-L achieves 60.9 mAP, which is slightly lower than SF-YOLO-L (63.7) despite having 1.9x fewer parameters. The authors attribute this to the gap in model capacity. A potential future direction is integrating knowledge distillation or larger YOLO variants to address capacity-constrained scenarios.
• Computational Overhead of MARD: Although MARD is only used during training, it increases the time overhead per epoch by 12.2% (+17s), which could present a bottleneck on extremely large-scale datasets. Future research could explore sparser sampling strategies or lower-frequency MARD evaluations (currently calculated every \(I\) steps).

Related Work & Insights

• vs. SF-YOLO (ECCV'24 Workshop): Both implement SFOD within the YOLO family. However, SF-YOLO is built upon the NMS-dependent single-head YOLOv5, which bypasses the dual-head complementarity issue and overlooks feature rank degradation under domain shift. In contrast, this work is the first systematic exploration of the newer NMS-free dual-head paradigm, with DHF providing a source of gain inaccessible to SF-YOLO.
• vs. FALCON-SFOD (CVPR'26): FALCON leverages foundation model (such as CLIP/DINO) priors to boost target localization in representations, essentially relying on plug-in external knowledge. In contrast, the proposed MARD recovers feature discriminability from within the detector through regularization, requiring no external models, rendering it more lightweight and orthogonal (hence stackable) to FALCON.
• vs. VICReg (ICLR'22): The formulation of MARD's variance-covariance regularization draws inspiration from VICReg, but with critical distinctions: (a) VICReg operates on global embeddings, whereas MARD operates on detection PAN features; (b) VICReg utilizes uniform sampling, while MARD employs pseudo-box-guided detection-aware sampling; (c) MARD introduces scale allocation and adaptive weighting mechanisms. This underscores that adapting general regularization techniques from representation learning into specific task architectures can yield substantial benefits.

Rating

• Novelty: ⭐⭐⭐⭐ The first systematic study of SFOD for NMS-free dual-head detectors. Both the "anchor-gapfiller" fusion strategy of DHF and the quantification of feature domain shift via effective rank offer fresh perspectives, though the core modules (MT and variance-covariance regularization) themselves are not entirely novel concepts.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluation across four domain shift benchmarks, four model scales, and five additional dual-head detectors confirms generalization. Comprehensive analyses include ablation studies, hyperparameter sensitivity, and training overhead. The multi-dimensional efficiency evaluation spanning FPS, parameter size, and latency is rare in typical SFOD papers and highly commendable.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivation is exceptionally clear (with quantitative evidence of the two bottlenecks intuitively shown in Fig. 2), the methodology is rigorous (with exact formulas, thresholds, and sampling strategies defined), the experimental analyses are insightful (explaining underlying reasons rather than just reporting numbers), and the supplementary materials are comprehensive (covering failure analysis, cold-start robustness, and cross-architecture migration).
• Value: ⭐⭐⭐⭐ This work pushes forward the Pareto frontier of accuracy, speed, and size in real-time SFOD. The proposed method is simple and generalizable, and the codebase is open-source, carrying direct practical value for deployment-sensitive real-time scenarios like autonomous driving. Since both modules are training-time-only, they present zero inference overhead, making it highly engineering-friendly.

Related Papers

• [ECCV 2026] LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection
• [ECCV 2026] PLOT: Pseudo-Labeling via Object Tracking for Monocular 3D Object Detection
• [ICCV 2025] RTMap: Real-Time Recursive Mapping with Change Detection and Localization
• [ICLR 2026] SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms
• [ICCV 2025] DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmetic