Domain Adaptive Object Detection via Dual-Stream Bilevel-Cycle Optimization¶
Conference: ECCV 2026
Paper: ECCV Proceedings
Area: Object Detection
Keywords: domain adaptive object detection, cycle self-training, bilevel optimization, pseudo-label, Mean Teacher
TL;DR¶
DSBCO splits cycle self-training (CST) into two cycle streams — one for classification and one for regression — and builds them on top of Mean Teacher, replacing the unstable sample-kernel inversion with a feature-dimension ridge-regression projection that is verified on the source domain, plus a normalization of regression offsets; it lifts the FCOS self-training baseline from 41.2 to 64.7 mAP on Cityscapes→Foggy Cityscapes.
Background & Motivation¶
Domain adaptive object detection (DAOD) must transfer a detector to a target domain with only source-domain annotations, and self-training is one of the dominant recipes: a model trained on the source domain labels the target domain, and the model is then retrained iteratively on source ground truth plus target pseudo-labels. Because of the domain shift, target pseudo-labels are unreliable by construction, and once they drift, the error is amplified across iterations — standard self-training therefore has a clearly visible accuracy ceiling that further iterations cannot break. Cycle Self-Training (CST) offered another route in image classification: it relaxes the assumption that source and target share one classifier, introduces a dedicated target-domain classifier, and constrains it with an inner/outer loop, thereby mitigating pseudo-label unreliability. Extending CST to object detection, however, had no workable solution before this paper.
Through experiments and theoretical analysis, the paper identifies three concrete obstacles. ❶ Unreliable pseudo-labels in detection take three forms — wrong classification only, wrong localization only, and both at once; the ceiling of standard self-training is set jointly by all three, whereas classification only faces the first. ❷ Detection is dense prediction: one image contains several objects and every location emits both a classification and a regression output, so dropping CST onto a one-stage detector such as FCOS makes training extremely unstable, up to model collapse. ❸ Regression targets are unbounded box offsets whose numerical range is far wider than softmax-bounded class probabilities; applying cycle consistency directly to regression makes the loss explode, and Mean Teacher without normalization produces severe box shifts and model collapse.
The paper's angle is to extend CST's two core ingredients — the inner loop learning on the target domain and the outer loop verifying back on the source domain — to classification and regression simultaneously, and to move the stability problem from the algorithm level to the optimization level: the whole framework sits on Mean Teacher, whose teacher is an exponential moving average (EMA) of the student and only consumes weakly augmented target images, using a temporal ensemble to suppress the variance of dense prediction; the regression stream standardizes regression offsets by target-domain statistics before entering the cycle. Core idea: formulate domain adaptive detection as a bilevel optimization problem — the inner loop solves a linear projection from target features to teacher pseudo-labels on the target domain, and the outer loop applies that same projection to source features and requires it to reproduce source ground truth, so that source-domain verification rejects pseudo-solutions supported by background noise; classification and regression each get such a cycle stream, and the regression stream adds normalization to tame unbounded coordinates.
Method¶
Overall Architecture¶
DSBCO is built on FCOS (anchor-free, with classification and regression decoupled into two per-pixel prediction heads) with a VGG-16 backbone, and the pipeline runs left to right and closes once per training iteration. The inputs are source-domain and target-domain images: the teacher receives only weakly augmented (resize, horizontal flip) target images and outputs pseudo-labels carrying class probability, localization quality (centerness) and regression offsets; the student is the only trainable model and receives strongly augmented source images (color jittering, Gaussian blur, random erasing) and strongly augmented target images, first trained on source ground truth in a supervised way (Focal classification loss + GIoU regression loss). The second stage is the baseline self-training, i.e. Instance Consistency Distillation: a binary mask is built from dual thresholds on the teacher's class probability and centerness, and the student's predictions are pulled toward the teacher's soft labels only at the high-confidence locations the mask keeps. The third stage is the paper's core, Feature Bilevel-Cycle Optimization: after burn-in, each iteration solves an explicit linear projection from the student's target-domain features (the inner loop, Inner Ridge) and then applies that projection directly to source features, requiring it to reproduce source ground truth (the outer loop, Outer Projection); classification and regression each get such a stream, and the regression stream is normalized before entering the cycle. The three losses are summed and back-propagated to update the student, the teacher follows via EMA, and in the next round the teacher has changed, the pseudo-labels have changed, and everything in the inner and outer loops is recomputed.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Strongly augmented source and target images"] --> B["Mean Teacher paradigm<br/>EMA teacher sees weakly augmented target only"]
B --> C["Instance Consistency Distillation<br/>dual-threshold mask on class and localization"]
C -->|enabled after burn-in| D["Classification cycle stream<br/>solve on target, verify on source"]
C -->|enabled after burn-in| E["Regression cycle stream<br/>normalize, then solve and verify"]
D --> F["Joint backward pass<br/>update student, EMA-update teacher"]
E --> F
F -.->|regenerate pseudo-labels next round| B
Key Designs¶
1. Mean Teacher paradigm: rescuing CST from training collapse with EMA
CST was designed for image classification, where both its inner and outer loops assume one prediction per sample; moved directly onto a dense-prediction detector, training oscillates violently or collapses outright. DSBCO does not alter the cycle structure to fix this — it first swaps in a stable optimization substrate. The student is trainable while the teacher is never updated by gradients but is an EMA of the student, \(\theta_{tea} \leftarrow \alpha \theta_{tea} + (1-\alpha)\theta_{stu}\), and the data split between the two networks is fixed: the teacher sees only weakly augmented target images, the student sees strongly augmented source and target images.
The reasoning is that dense prediction emits classification and regression outputs at thousands of locations per image at once, so the pseudo-labels produced by the student at the current step have enormous variance and using them as supervision targets amounts to self-training on noise; the EMA teacher integrates the supervision signal over time and weak augmentation removes additional input-side perturbation, so pseudo-label jitter is suppressed. FCOS rather than Faster R-CNN is chosen for what comes later: FCOS decouples classification and regression into two independent per-pixel heads, which is what allows the dual-stream cycle optimization to act on two sets of outputs without interference. This step only addresses whether training can stay stable, not whether the pseudo-labels are correct, so it must be used together with the cycle streams below.
2. Instance Consistency Distillation: filter samples by dual thresholds, then align soft labels
The teacher itself comes from a different domain, so its target-domain outputs are mixed with heavy background noise; enforcing consistency over the whole image would consolidate that background noise as supervision. DSBCO therefore builds a binary mask \(M = \mathbb{I}\left(\hat{p}_t > \tau_{cls} \land \hat{c}_t > \tau_{reg}\right)\) before distillation, keeping only locations whose class probability and localization quality (centerness) both exceed their thresholds, and computes the loss inside the valid region defined by that mask.
Inside the mask two channels are aligned. For classification, the teacher provides continuous soft labels, so a Quality Focal Loss pulls the student's prediction toward the teacher's soft label, with a modulating factor β that down-weights easy samples where the student already agrees with the teacher and concentrates gradient on locations not yet aligned. For regression, a GIoU loss measures the discrepancy between the student's predicted boxes and the teacher's pseudo-label boxes. The dual threshold is not an incidental engineering detail: it maps exactly onto the three failure forms of pseudo-labels in the motivation — class probability alone cannot block localization errors, centerness alone cannot block classification errors, and only gating on both dimensions at once screens out all three forms, which is where it is cleaner than conventional self-training that filters pseudo-labels by confidence alone.
3. Classification cycle stream: replacing CST's sample kernel with a feature-dimension ridge regression verified on the source domain
The inner loop of the original CST has to construct and invert a sample kernel matrix; with many samples the matrix becomes ill-conditioned and costly, and since it acts on image-level classification outputs it cannot exploit the feature space a detector has already decoupled. DSBCO instead builds the covariance along the feature dimension and solves a ridge regression with a squared-error plus L2 objective, obtaining the optimal linear projection \(W_{cls}\in\mathbb{R}^{d\times C}\) that maps the student's target-domain features \(Z_t \in \mathbb{R}^{N_t\times d}\) to the teacher's pseudo-label vectors \(\hat{p}_t\):
The outer loop (Outer Projection) then applies this explicit mapping head learned on the target domain directly to the source features \(Z_s\) and requires the projected result to match source ground truth, \(L_{cstcls} = \left\|Z_s W_{cls} - y_s\right\|_F^2\).
The crux is that this is a cycle of "learn the mapping on the target, verify it on the source." A mapping that has genuinely captured object semantics should also map source features onto source labels; conversely, if the mapping is only supported by target-domain background correlations — precisely the spurious solution that Theorem 2 says standard self-training converges to — it incurs a large reconstruction error on the source domain and is penalized directly by this term. The labeled source term thus acts as a free correctness checker, with no extra discriminator and no adversarial training. This is also the fundamental difference from global adversarial alignment of the DANN family: adversarial alignment only requires that the two domains' feature distributions "look alike" and is indifferent to whether what is being aligned is noise and background, whereas cycle alignment carries an explicit supervision direction and aligns classification and regression separately.
4. Regression cycle stream: standardize the offsets first, then enforce cycle consistency
Regression targets are unbounded box offsets whose numerical range is much wider than class probabilities; applying cycle consistency directly to regression explodes the loss and causes large box shifts. DSBCO adds a normalization layer inside the regression stream: the inner loop first computes the mean \(\mu_t\) and standard deviation \(\sigma_t\) of the teacher's target-domain regression offsets \(\hat{\delta}_t\) and standardizes them into a standard distribution, \(\tilde{\delta}_t = (\hat{\delta}_t - \mu_t)/(\sigma_t + \epsilon)\), then solves the regression projection \(W_{reg}\in\mathbb{R}^{d\times 4}\) in that standardized space; the outer loop standardizes the source ground truth with the same target-domain statistics to obtain \(\tilde{y}_s\) and computes \(L_{cstreg} = \left\|Z_s W_{reg} - \tilde{y}_s\right\|_F^2\).
Normalization is necessary rather than merely cosmetic: if the inner loop solves in the standardized space while the outer loop compares in the original coordinate space, the consistency constraint becomes a comparison between quantities in two different scales, which neither converges nor avoids amplifying the large gradients of unbounded coordinates straight back through the network. Once both sides live in the standard space, the regression cycle's gradient scale becomes comparable to the classification cycle's and training stays stable. The paper records this step as mapping large regression values into a stable space, and derives a generalization bound for the regression task in Theorem 1, showing that the target-domain localization error is controlled jointly by the source cycle reconstruction error, the student-teacher discrepancy and regression non-robustness (⚠️ the formulas are garbled in the cached text; refer to the original paper and Section A of the supplementary material for their exact form and constants).
A Worked Example¶
Take one iteration after burn-in has ended, and trace where the "cycle" actually closes:
- The teacher receives a weakly augmented target image and emits per-location class probabilities \(\hat{p}_t\), centerness \(\hat{c}_t\) and regression offsets \(\hat{\delta}_t\); most locations are judged low-confidence by the mask \(M\) and discarded, leaving only the few locations that pass both thresholds as valid supervision;
- The student consumes the strongly augmented source and target images and computes the supervised loss \(L_{sup}\) and the masked distillation loss \(L_{distill}\);
- Inner loop: taking the student's target-domain features \(Z_t\) (columns are features, rows are valid locations) together with the teacher's soft labels, a closed-form ridge regression solves for \(W_{cls}\) with a single \(d\times d\) matrix inversion; the regression stream standardizes the offsets with \(\mu_t,\sigma_t\) first and then solves \(W_{reg}\) the same way;
- Outer loop: both projections learned on the target domain are transplanted unchanged onto the source features \(Z_s\), the classification result is compared with the source labels and the regression result with the standardized source labels \(\tilde{y}_s\), yielding \(L_{cstcls}\) and \(L_{cstreg}\);
- The three losses are summed with weights and back-propagated to update the student; the teacher follows the student via EMA;
- Next iteration the teacher's parameters have changed → the pseudo-labels and the mask change → \(Z_t\), \(\mu_t\), \(\sigma_t\) and both projection matrices are all recomputed.
This loop is exactly "pseudo-labels → solve the mapping → verify on the source → update the model → generate new pseudo-labels": pseudo-labels are not passively accepted as supervision targets, they must pass the source-side verification each round before they count.
Loss & Training¶
The total objective sums the source-domain supervised loss, the instance-level consistency distillation loss and the feature-level cycle alignment loss:
Here \(L_{sup}\) consists of a Focal classification loss and a GIoU regression loss (GIoU rather than L1 avoids scale sensitivity and the vanishing gradient of standard IoU, which matters for the early stability of anchor-free models); \(L_{distill}\) averages the classification and regression consistency losses inside the region defined by the mask; \(\lambda_{unsup}\) and \(\lambda_{cycle}\) balance the unsupervised signals and \(\eta\) alone controls the relative weight of the regression cycle. Training runs in three phases: 30k supervised pre-training iterations on the source domain followed by 90k adaptation iterations, SGD at a learning rate of 0.004 on a single RTX 3090; in the self-training baseline the pretrained detector filters predictions with confidence below 0.45; the bilevel cycle optimization only starts after \(t > T_{burn}\) (the teacher is first stabilized by distillation before the cycle is switched on). The best hyper-parameters found by ablation are \(\lambda_{cycle}=0.5\), \(\lambda_{unsup}=0.01\) and \(\eta=0.01\).
Key Experimental Results¶
Main Results¶
Four standard cross-domain scenarios; the metric is mAP at IoU=0.5 (Cityscapes / Foggy Cityscapes / BDD100K report all shared categories, while KITTI→Cityscapes and Sim10K→Cityscapes report the AP of the Car category):
| Scenario | Dataset (source → target) | Metric | DSBCO | Best prior FCOS method | Gain |
|---|---|---|---|---|---|
| Adverse weather | Cityscapes → Foggy Cityscapes | mAP | 64.7 | HT 50.4 | +14.3 |
| Synthetic source | Sim10K → Cityscapes | Car AP | 68.6 | HT 65.5 | +3.1 |
| Distinct camera | KITTI → Cityscapes | Car AP | 62.2 | HT 60.3 | +1.9 |
| Diverse context | Cityscapes → BDD100K | mAP | 41.9 | HT 40.2 | +1.7 |
The same tables include a set of non-FCOS detectors as reference (AT+REACT and CMT+DSD-DA on Faster R-CNN, MRT and MTM on Def DETR, MGCAMT on RetinaNet). On Foggy Cityscapes, for instance, DSBCO's 64.7 mAP is 8.8 above the second-best method overall (MGCAMT, 55.9) and 14.3 above the second-best FCOS-based method (HT, 50.4); per class the gains concentrate on the hard categories: Truck 58.1 (HT 32.7), Train 62.2 (HT 49.1), Motor 58.8 (HT 40.1), while the gains on dominant classes such as Person and Car are milder.
One caveat deserves stating plainly: the paper's claim of exceeding "all existing FCOS detector methods" is, strictly speaking, confined to FCOS detectors. Across detectors, MGCAMT (RetinaNet) reaches 44.8 on BDD100K against this paper's 41.9, and matches 62.2 on KITTI→Cityscapes; DSBCO leads the FCOS baselines consistently across all four scenarios but is not the overall best under every detector comparison.
Cost of the bilevel optimization (Table 5; \(N\) is the number of samples, \(T\) the unit training time of standard ST, and \(M\) the number of inner gradient steps of naive bilevel optimization, typically \(M\approx50\)):
| Optimization strategy | Complexity | Time cost |
|---|---|---|
| Standard ST | \(O(N)\) | \(T\) |
| Naive bilevel optimization | \(O(N\cdot M)\) | \(\approx M\cdot T\) |
| DSBCO (Ours) | \(O(N)\) | \(\approx 1.2\cdot T\) |
Naive bilevel optimization is unusable for detection because dense box samples repeatedly construct large Hessians, which is both slow and unstable; DSBCO replaces sample-kernel inversion with a compact feature-dimension covariance estimate and, because the ridge regression has a closed-form solution, skips the iterative inner solve entirely, costing only about 20% more training time than standard self-training.
Ablation Study¶
Incremental ablation on Cityscapes → Foggy Cityscapes (mAP %):
| Config | \(L_{distill}\) | \(L_{cstcls}\) | \(L_{cstreg}\) | mAP |
|---|---|---|---|---|
| Standard ST | 41.2 | |||
| + Instance Consistency | ✓ | 48.3 | ||
| + Classification Cycle | ✓ | ✓ | 54.5 | |
| + Regression Cycle | ✓ | ✓ | 50.9 | |
| DSBCO (full) | ✓ | ✓ | ✓ | 64.7 |
Hyper-parameter sensitivity (Figs. 4 and 5): the optimum is reached at \(\lambda_{cycle}=0.5\), \(\lambda_{unsup}=0.01\) and \(\eta=0.01\), and the best base learning rate is 0.004.
Key Findings¶
- Instance consistency distillation is the foundation that contributes most: 41.2 → 48.3 (+7.1), showing that "dual-threshold sample filtering + soft-label alignment" is by itself substantially better than naive self-training, consistent with the motivation that the ceiling of standard ST is mainly set by unreliable pseudo-labels.
- The two cycle streams contribute asymmetrically and are markedly super-additive in combination: adding only the classification cycle gives 54.5 (+6.2), adding only the regression cycle gives 50.9 (+2.6), while both together give 64.7, which is 10.2 above the best single-stream configuration and far more than the sum of the individual gains. Semantic alignment appears to be the main source of benefit, and the regression cycle is of limited use alone but lifts the classification stream further when present. ⚠️ The paper does not explain this super-additivity; my own reading is that the regression stream's standard space only becomes genuinely identically distributed once the classification stream has first pulled intra-class features together, so the two are mutually preconditioning — this is my inference, not a conclusion of the paper.
- The scenario spread is informative: the largest gain is on adverse weather (+14.3) and the smallest on BDD100K (+1.7). Degradation on Foggy Cityscapes is dominated by visibility-induced localization drift, precisely what the regression cycle targets, whereas the BDD100K domain gap stems more from scene layout and geography and cannot be solved by standardizing a coordinate space alone. Even the synthetic-to-real texture gap of Sim10K→Cityscapes yields +3.1, so the method is not tied to a single kind of domain shift.
- Feature-space visualization: under t-SNE, DSBCO's clusters are tighter and the boundaries cleaner than standard ST (Person and Rider cluster more tightly, and the separations between Bicycle/Motor and between Train/Bus are more pronounced), which is consistent with the claim that the cycle alignment pulls semantic features together.
- Failure cases follow clear patterns: missed distant cars at night under low visibility, missed nearby persons under daytime motion blur, missed nearby cars during fast nighttime driving with severe blur, and a nighttime case where Train is misclassified as Car. The first three combine small objects with imaging degradation, and the last shows that inter-class semantics still bleed under extremely low illumination.
Highlights & Insights¶
- "Learn the mapping on the target, verify it on the source" is the key abstraction for porting CST to detection: it turns pseudo-label selection into a verification problem with a closed-form solution — source annotations themselves are the checker, so no extra discriminator or adversarial training is needed and there is no fragile adversarial balancing coefficient to tune. Any task with labeled source data and unlabeled target data over structured or continuous outputs can borrow this shape.
- Replacing the sample kernel with a feature-dimension covariance kills two birds: the sample kernel of the original CST grows with the number of samples and easily becomes ill-conditioned, whereas in the feature dimension the matrix size depends only on the (fixed) number of feature channels; combined with the closed-form ridge solution, this pulls naive bilevel optimization from \(O(N\cdot M)\) back to \(O(N)\) and costs only about 20% more training time. It is a textbook engineering trade-off that makes bilevel optimization usable rather than fancier.
- Regression normalization is the precondition that makes a cycle valid on regression, not a finishing touch: class probabilities are bounded by softmax while box offsets are unbounded, and when the two sides are not on the same scale the consistency constraint amplifies large gradients straight back through the network. The observation is simple but easy to overlook, and the paper pairs it with Theorem 1's regression bound showing how the three error terms are controlled.
- The theory gives a positive explanation of why standard ST fails (Theorem 2: when background correlations dominate the pseudo-labels, standard ST converges to a spurious solution and the target error has an irreducible lower bound ε) rather than only proving a bound for its own method; this motivates why source verification can reject spurious solutions (Theorem 3), so the theoretical narrative and the design interlock. ⚠️ The formulas are badly garbled in the cached text; refer to the original paper and supplementary material for their exact form.
- Transferable design: any pseudo-label iterative task (semi-supervised segmentation, self-training keypoint detection, domain adaptive depth estimation) can adopt "solve a lightweight explicit mapping on the target domain in the inner loop + verify on a labeled domain in the outer loop", at the cost of one \(d\times d\) inversion per step, usually far cheaper than the backward pass itself.
Limitations & Future Work¶
- Limitations admitted by the authors: extending this dual-stream optimization to modern architectures such as Transformers requires further study; the fixed thresholding mechanism may struggle against complex background noise in challenging scenarios, prompting future work on adaptive strategies.
- Limitations I spot: experiments use only one backbone, FCOS with VGG-16, with no comparison against ResNet or Transformer backbones, so it is hard to tell how much of the gain comes from the backbone choice; the cross-detector claim is more confident than the numbers (BDD100K and KITTI→Cityscapes are not overall best); the theory states an upper bound and convergence properties but no experiment checks whether the bound is tight or measures the individual error terms; and the efficiency comparison reports only training time and complexity order, not memory or inference cost, while the per-step matrix inversion inside the inner loop has a constant cost that the \(O(N)\) notation hides when \(d\) is large.
- Concrete improvement directions: replace the fixed thresholds with uncertainty-based adaptive filtering so the mask is neither too loose nor too tight in background-heavy scenarios such as nighttime BDD100K; extend the dual streams to the DETR family while noting that DETR predictions are not decoupled per pixel, so the inner-loop form of \(W\) must be redesigned; and add a per-class AP comparison that separates localization gains from classification gains, which would also test statements such as the claim that gains on rigid structures like Bus come from the regression cycle.
Related Work & Insights¶
- vs CST (Liu et al., NeurIPS 2021): the original CST relaxes the shared-classifier assumption at image level and uses an inner/outer loop to mitigate pseudo-label unreliability. This paper needs two changes to make it apply to detection — replacing the unstable sample-kernel inversion with a feature-dimension covariance and a closed-form ridge solution (cutting cost from \(O(N\cdot M)\) to \(O(N)\)), and adding a normalized regression stream — which moves CST from the classifier level down to the feature-space level.
- vs Mean Teacher and consistency-distillation methods (Tarvainen & Valpola; AT, CMT, etc.): these perform instance-level consistency only, essentially trusting the teacher's soft labels unconditionally (or filtering by confidence alone), so their ceiling is still set by pseudo-label quality; DSBCO treats Mean Teacher purely as a stable substrate and stacks a feature-level cycle loss with source verification on top, which is exactly the 48.3 → 64.7 gap in the ablation.
- vs DANN and adversarial feature alignment: adversarial alignment makes a discriminator unable to tell the domains apart, an indiscriminate global constraint that aligns noise and background along with everything else; cycle alignment here has an explicit supervision direction (the mapping must reproduce labels on the source domain) and aligns classification and regression in separate streams, so the alignment objective cannot be dominated by the regression quantities.
- vs HT (Harmonious Teacher), SIGMA, SIGMA++: these also build on FCOS but follow teacher harmonization and semantic graph matching; DSBCO's advantage over them shows up mainly on the hard classes of Foggy Cityscapes (Truck 58.1 vs HT's 32.7, Motor 58.8 vs 40.1), suggesting that "treating pseudo-labels as supervision targets" and "treating pseudo-labels as mappings to be verified" are two different sources of gain.
Rating¶
- Novelty: ⭐⭐⭐⭐ Genuinely porting image-level CST to detection with the necessary dual-stream and normalization changes; the "verify the projection on the source domain" idea is more substantial than simply adding another consistency loss.
- Experimental Thoroughness: ⭐⭐⭐ Four benchmarks plus incremental ablation plus an efficiency comparison is adequate coverage, but only one backbone is tested, there is no cross-backbone control, and the theoretical bound is never validated empirically.
- Writing Quality: ⭐⭐⭐ The logic chain from motivation to method is clear and the four theorems and the ablation are spelled out; the weaknesses are claims about cross-detector superiority that are more confident than the numbers and a missing link between theory and experiments.
- Value: ⭐⭐⭐⭐ Provides a reusable, cost-controlled template for "bilevel/cycle self-training for dense prediction" and exposes how standard self-training fails in detection, both directly useful to follow-up DAOD work.