RT-SDGOD: Real-Time Single-Domain Generalized Object Detection¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: real-time object detection, single-domain generalization, multi-evidence collaborative modeling, DETR, domain shift
TL;DR¶
Addressing the issue that real-time detectors suffer severe degradation dominated by missed detections under adverse weather and lighting shifts, this paper proposes RT-SDGDet, a zero-inference-overhead framework that expands and stabilizes object-level discriminative evidence via one-to-many query grouping, deformable attention diversity regularization (DEDL), and dual-view consistency alignment (DvECL).
Background & Motivation¶
In latency-sensitive real-world computer vision systems such as autonomous driving and intelligent surveillance, object detectors must operate reliably under strict constraints on inference latency, model footprint, and computational budget. Although modern YOLO variants and real-time DETRs have continuously pushed the boundary of speed-accuracy trade-offs on in-domain benchmarks, real-world variations in illumination, weather, and imaging conditions inevitably introduce severe distribution shifts that drastically degrade detection performance. Conventional Single-Domain Generalized Object Detection (SDGOD) methods predominantly rely on two-stage architectures, test-time adaptation (TTA), or heavy auxiliary feature disentanglement branches. These designs inevitably incur considerable inference overhead and runtime latency, rendering them impractical for real-time edge deployment. To bridge this critical gap, this work formalizes a stricter and more deployment-faithful problem setting termed Real-Time Single-Domain Generalized Object Detection (RT-SDGOD). In this paradigm, a detector is trained strictly on labeled data from a single source domain and evaluated across diverse unseen target domains, while the test-time inference pipeline remains completely untouched—disallowing any runtime adaptation, parameter updates, or auxiliary computational branches. Consequently, cross-domain robustness must stem entirely from training-time representation learning.
To uncover why Transformer-based real-time detectors collapse under environmental shifts, the authors conducted controlled error-pattern attribution and sensitivity analyses on set-prediction detectors. Under progressively intensified weather and lighting perturbations, class confusion rates remain remarkably low and stable, while false positive rates (false alarms) even decline slightly. Instead, the dominant driver of performance degradation is a steep and consistent surge in false negatives (missed detections). Further Grad-CAM response inspections and local occlusion/blurring diagnostics reveal that high-confidence predictions in real-time DETRs heavily depend on an extremely narrow set of local discriminative cues rather than the comprehensive structure of the target object. When adverse weather obscures these fragile cues or imaging artifacts drift local representations, prediction confidence drops precipitously, demonstrating that existing detectors suffer from severely constrained evidence coverage and fragile activation stability.
Addressing this vulnerability requires moving beyond coarse global data augmentation or synthetic domain generation, which fail to rectify evidence concentration at the object level. The core challenge in RT-SDGOD is to steer the detector during training away from fragile local shortcuts, encouraging multiple queries of the same instance to collaboratively mine complementary evidence across the target and preserve cross-view consistency under appearance perturbations. Core Idea: Construct exclusive object-specific query groups via one-to-many (O2M) supervision, build explicit spatial evidence descriptors from deformable cross-attention, enforce complementary coverage across intra-group queries via Discriminative Evidence Diversity Learning (DEDL), and align cross-view representations using Dual-view Evidence Consistency Learning (DvECL) with mutual nearest neighbors, achieving omission-resistant domain generalization with zero extra inference overhead.
Method¶
Overall Architecture¶
RT-SDGDet is built upon the real-time detection transformer RF-DETR, comprising a backbone and multi-scale encoder for feature extraction, a multi-layer Transformer decoder for iterative query updating, and prediction heads that output classification logits and bounding boxes without non-maximum suppression (NMS). During test-time inference, the architecture remains strictly identical to standard RF-DETR using standard one-to-one (O2O) prediction, incurring zero additional FLOPs, parameters, or latency. During training, the framework introduces a dual-view input pipeline alongside three synergistic representation learning mechanisms: first, an exclusive one-to-many (O2M) label assignment constructs stable target-specific query groups; second, at every decoder layer, explicit evidence descriptors are derived from deformable cross-attention to penalize spatial overlap among intra-group queries via Discriminative Evidence Diversity Learning (DEDL); third, query correspondences between original and augmented views are established based on descriptor similarity to enforce representation invariance via Dual-view Evidence Consistency Learning (DvECL).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dual-view Input<br/>Source View + Augmented View"] --> B["Multi-scale Encoder<br/>Multi-scale Feature Extraction"]
B --> C["O2M Exclusive Query Grouping<br/>IoU and Classification Quality Matrix"]
C --> D["Discriminative Evidence Diversity Learning (DEDL)<br/>Deformable Cross-Attention Descriptors + Margin Penalty"]
D --> E["Dual-view Evidence Consistency Learning (DvECL)<br/>Mutual Nearest Neighbor Matching + Cosine Alignment"]
E --> F["Joint Training Optimization / Clean Inference<br/>Zero Runtime Overhead"]
Key Designs¶
1. Exclusive Object-Specific Query Grouping via O2M Supervision: Establishing Stable Carriers for Multi-Evidence Modeling Standard DETR training enforces one-to-one (O2O) bipartite matching, allocating only a single positive query per ground-truth object, which yields sparse positive gradients and prevents a single instance from distributing its learning across multiple pieces of evidence. While one-to-many (O2M) supervision introduces multiple positive queries per instance, conventional matching allows queries to oscillate ambiguously across overlapping targets. To provide a stable foundation for multi-evidence collaboration, the authors refine the matching procedure. At decoder layer \(l\), a query-target quality matrix is computed using predicted box IoU and classification confidence: $\(T_{nq}^{(l)} = \lambda_{\mathrm{box}} \operatorname{IoU}(b_q^{(l)}, \bar{b}_n) + \lambda_{\mathrm{cls}} p_q^{(l)}(\bar{k}_n)\)$ where \(\bar{b}_n\) and \(\bar{k}_n\) denote the ground-truth box and category for object \(n\), and \(p_q^{(l)}(\bar{k}_n)\) is the predicted probability. To prevent ambiguous competition, the assignment initially guarantees coverage by allocating an unused query to each target, then greedily incorporates high-quality queries under an exclusivity constraint up to a cap of \(W\) queries per object. Merging the O2O-matched query yields the object-specific group \(\mathcal{G}_n^{(l)}\), ensuring that all constituent queries unambiguously supervise the same physical target.
2. Discriminative Evidence Diversity Learning (DEDL): Enforcing Complementary Spatial Evidence Coverage Across Queries To eliminate the detector's over-reliance on isolated, fragile local cues, intra-group queries must be guided toward distinct, complementary regions of the target. Directly regularizing deep query embeddings risks corrupting categorical and coordinate semantics. Instead, DEDL constructs explicit evidence descriptors directly from the query-wise deformable cross-attention module, where attention weights indicate feature reliance strength and sampling offsets determine physical spatial locations. Averaging across attention heads and concatenating flattened weights with weighted coordinates yields the normalized evidence descriptor: $\(\mathbf{E}^{(l)} = \operatorname{Norm}\left(\left[\mathbf{w}^{(l)},\; \mathbf{w}^{(l)} \odot \mathbf{x}^{(l)},\; \mathbf{w}^{(l)} \odot \mathbf{y}^{(l)}\right]\right)\)$ For each object-specific group with normalized descriptors \(\{\mathbf{e}_{n,i}^{(l)}\}_{i=1}^{S_n}\), DEDL imposes a pairwise inner-product penalty governed by a similarity margin \(m\): $\(\mathcal{L}_{\mathrm{div}}^{(l)} = \frac{1}{N_l} \sum_{n=1}^{N_l} \frac{1}{S_n(S_n - 1)} \sum_{i \neq j} \max\left(0, \langle \mathbf{e}_{n,i}^{(l)}, \mathbf{e}_{n,j}^{(l)} \rangle - m\right)\)$ Summed across all decoder layers, this diversity loss penalizes redundant spatial attention, compelling queries within the same group to attend to complementary components (such as a vehicle's headlights, wheels, roofline, and windshield). If adverse weather degrades one local cue, the remaining queries still provide sufficient evidence to maintain activation.
3. Dual-view Evidence Consistency Learning (DvECL): Cross-View Mutual Nearest Neighbor Pairing and Robust Representation Alignment While DEDL promotes functional specialization across queries, such localized evidence remains susceptible to feature drift under severe appearance shifts. DvECL reinforces representation stability across views by pairing and aligning queries that capture identical evidence. During training, each training image is paired with an augmented view generated via Multi-view Adversarial Discriminator (MAD) perturbations. Because query order and group size differ across views, naive index-based alignment fails. Instead, cross-view correspondences are determined by evidence descriptor similarity: $\(\mathbf{M}_n^{(l)}(i,j) = \langle \mathbf{e}_{n,i,\mathrm{src}}^{(l)}, \mathbf{e}_{n,j,\mathrm{aug}}^{(l)} \rangle\)$ A mutual nearest neighbor criterion filtered by a similarity threshold \(\tau\) retains only query pairs \((i, j)\) that are reciprocally most similar and satisfy \(\mathbf{M}_n^{(l)}(i,j) > \tau\), forming the matched set \(\mathcal{P}^{(l)}\). A cosine distance loss \(\mathcal{L}_{\mathrm{cons}}^{(l)}\) is then minimized between the matched query feature vectors \(\mathbf{h}_{n,i,\mathrm{src}}^{(l)}\) and \(\mathbf{h}_{n,j,\mathrm{aug}}^{(l)}\) across all decoder layers. This anchors the semantic stability of specific physical evidence patterns across corruptions, mitigating domain-induced representation shifts.
Loss & Training¶
The overall training objective combines the primary detection losses with the auxiliary regularization terms: $\(\mathcal{L} = \mathcal{L}_{\mathrm{O2O}} + \lambda_{\mathrm{O2M}} \mathcal{L}_{\mathrm{O2M}} + \lambda_{\mathrm{DEDL}} \mathcal{L}_{\mathrm{DEDL}} + \lambda_{\mathrm{DvECL}} \mathcal{L}_{\mathrm{DvECL}}\)$ where \(\mathcal{L}_{\mathrm{O2O}}\) and \(\mathcal{L}_{\mathrm{O2M}}\) comprise standard Focal classification loss and L1/GIoU bounding box regression losses. Training is conducted on 8 NVIDIA RTX 3090 GPUs with distributed data parallel processing using RF-DETR-L as the baseline. A batch size of 2 per GPU with 8-step gradient accumulation results in an effective batch size of 128. Optimization uses AdamW with initial learning rates of \(1 \times 10^{-4}\) for the detector and \(1.5 \times 10^{-4}\) for the encoder, a weight decay of \(1 \times 10^{-4}\), and gradient clipping at a maximum norm of 0.1. Training spans 48 epochs, decaying the learning rate at epoch 11. Key hyperparameters are configured as \(\lambda_{\mathrm{box}} = 0.7\), \(\lambda_{\mathrm{cls}} = 0.3\), group size cap \(W = 6\), similarity margin \(m = 0.3\), matching threshold \(\tau = 0.8\), and loss balancing weights \(\lambda_{\mathrm{O2M}} = 0.5\), \(\lambda_{\mathrm{DEDL}} = 0.5\), and \(\lambda_{\mathrm{DvECL}} = 0.3\).
Key Experimental Results¶
Main Results¶
Experiments are evaluated on the established SDGOD benchmark covering five driving scenarios. Daytime-Clear serves as the single source domain (19,395 training images and 8,313 test images), while the remaining four conditions (Night-Clear, Daytime-Foggy, Dusk-Rainy, and Night-Rainy) act as unseen target domains. All models are evaluated using mean Average Precision at 50% IoU (mAP@50).
| Method | Params (M) | GFLOPs | Daytime-Clear (Source) | Night-Clear | Daytime-Foggy | Dusk-Rainy | Night-Rainy | Avg. mAP |
|---|---|---|---|---|---|---|---|---|
| YOLOv12-L | 26.4 | 88.9 | 62.7 | 48.1 | 43.0 | 40.1 | 22.4 | 43.3 |
| YOLOv13-L | 27.6 | 88.4 | 61.8 | 47.3 | 42.2 | 40.4 | 23.1 | 43.0 |
| LW-DETR-M | 28.2 | 42.8 | 61.1 | 49.1 | 42.0 | 47.1 | 29.0 | 45.7 |
| D-FINE-L | 31.0 | 91.0 | 63.1 | 49.4 | 40.0 | 43.2 | 23.2 | 43.8 |
| DEIM-L | 31.0 | 91.0 | 64.1 | 50.3 | 43.0 | 43.6 | 24.8 | 45.2 |
| DEIMv2-L | 32.0 | 96.0 | 63.9 | 52.1 | 43.4 | 51.0 | 32.9 | 48.7 |
| RT-DETRv3-R34 | 31.0 | 92.0 | 62.8 | 46.7 | 40.1 | 39.1 | 16.9 | 41.1 |
| RT-DETRv4-L | 31.0 | 91.0 | 65.9 | 52.2 | 43.5 | 47.7 | 27.5 | 47.4 |
| RF-DETR-L (Baseline) | 33.9 | 125.6 | 68.6 | 56.4 | 48.2 | 52.6 | 33.5 | 51.9 |
| + ABA | 33.9 | 125.6 | 68.0 | 56.7 | 48.3 | 55.0 | 37.6 | 53.1 |
| + NP | 33.9 | 125.6 | 68.8 | 57.2 | 48.6 | 55.0 | 36.5 | 53.2 |
| + MAD | 33.9 | 125.6 | 68.7 | 57.1 | 48.9 | 54.7 | 36.9 | 53.3 |
| + OA-DG | 33.9 | 125.6 | 68.3 | 56.7 | 48.6 | 54.8 | 34.1 | 52.5 |
| + SRA | 33.9 | 125.6 | 68.3 | 56.2 | 49.9 | 54.7 | 36.0 | 53.0 |
| + PhysAug | 33.9 | 125.6 | 68.4 | 56.9 | 48.1 | 55.2 | 37.4 | 53.2 |
| RT-SDGDet (Ours) | 33.9 | 125.6 | 68.9 | 58.0 | 49.9 | 56.2 | 39.0 | 54.4 |
Ablation Study¶
The progressive contribution of each architectural component and objective is evaluated across all five domains:
| Config | D-Clear (Source) | N-Clear | D-Foggy | D-Rainy | N-Rainy | Avg. mAP | Note |
|---|---|---|---|---|---|---|---|
| RF-DETR-L (Baseline) | 68.6 | 56.4 | 48.2 | 52.6 | 33.5 | 51.9 | Base single-domain model |
| + O2M | 68.6 | 56.7 | 48.7 | 53.9 | 34.4 | 52.5 | Denser positive supervision stabilizes modeling (+0.6%) |
| + O2M + DEDL | 68.9 | 57.5 | 49.5 | 54.9 | 38.2 | 53.8 | Evidence diversity brings massive gains in harsh domains (+1.3%) |
| + O2M + DEDL + DvECL (Full) | 68.9 | 58.0 | 49.9 | 56.2 | 39.0 | 54.4 | Cross-view alignment achieves best overall robustness (+0.6%) |
In terms of False Negative Rate (FNR), RT-SDGDet reduces average miss rates across all scenarios from 18.5% down to 15.7% (with D-Foggy dropping from 28.6% to 24.5% and N-Rainy dropping from 26.7% to 21.6%), providing direct quantitative confirmation that multi-evidence learning mitigates omission-dominated degradation.
Key Findings¶
- Disproportionate Gains in Severely Degraded Domains: RT-SDGDet achieves its largest performance margins under extreme weather shifts, improving Night-Rainy and Dusk-Rainy by +5.5% and +3.6% mAP over baseline, compared to +1.6% on Night-Clear and +1.7% on Daytime-Foggy. This confirms that broadening evidence coverage is vital when primary local cues are severely corrupted.
- Synergistic Modular Progression: O2M establishes the candidate query pool (+0.6%); DEDL activates query specialization to drive the single largest boost (+1.3%, including +3.8% on Night-Rainy); DvECL locks in representation stability across corruptions (+0.6%).
- Moderate Hyperparameter Robustness: The margin \(m\) achieves peak performance at 0.3 across the interval \([0.0, 0.5]\). Setting regularization weights too high (\(\lambda_{\mathrm{DEDL}} > 0.8\) or \(\lambda_{\mathrm{DvECL}} > 0.5\)) slightly interferes with primary detection tasks, confirming that balanced weights (0.5 and 0.3) provide optimal trade-offs.
Highlights & Insights¶
- Practical Real-Time Problem Formulation: By enforcing zero runtime computational and structural overhead, this work addresses the key deployment barrier of prior SDGOD methods, making single-domain generalization viable on real-time embedded edge hardware.
- Attention-Grounded Evidence Descriptors: Extracting spatial descriptors directly from deformable cross-attention weights and sampling coordinates bypasses entangled semantic query vectors, offering an interpretable and mathematically grounded mechanism for diversity and alignment.
- Architecture-Agnostic Transferability: The paradigm of decomposing positive queries to supervise distinct physical regions can be readily integrated into any query-based detector or instance segmenter without altering its inference mechanics.
Limitations & Future Work¶
- Dependency on Deformable Attention Mechanisms: The evidence descriptor formulation relies directly on deformable cross-attention sampling points and weights, requiring non-trivial adaptations if applied to dense global-attention DETRs or CNN-based YOLO models.
- Increased Training Overhead: While inference cost is zero, forward-passing paired dual-view images and performing pairwise descriptor similarity matching across decoder layers increases GPU memory footprint and training wall-clock time.
- Future Directions: Exploring feature-space perturbation to eliminate the dual-image forward pass during training, and extending multi-evidence collaboration to real-time 3D perception and multimodal detection pipelines.
Related Work & Insights¶
- vs DG-DETR / SA-DETR: Prior DETR-based SDGOD approaches introduce auxiliary disentanglement networks or dynamic test-time updates that substantially inflate inference latency; RT-SDGDet confines all operations to training-time loss regularization, preserving exact real-time inference speed.
- vs Domain Augmentation Approaches (MAD / NP / PhysAug): Global and frequency-based augmentations manipulate image-level or channel-level statistics without resolving the detector's localized evidence concentration; RT-SDGDet explicitly distributes attention across object parts, offering complementary and strictly superior performance.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Introduces the RT-SDGOD setting and designs an elegant deformable attention-based evidence diversity and consistency mechanism]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across 5 driving domains, 15 real-time detectors and augmentation baselines, with in-depth error attribution and ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical definitions, cohesive narrative, and well-designed visualizations]
- Value: ⭐⭐⭐⭐⭐ [Directly tackles real-world autonomous driving edge deployment challenges with zero-inference-overhead generalization]