Robust Onion: Peeling Open Vocab Object Detectors Under Noise¶

Conference: ECCV 2026
Paper: ECCV 2026 official page
Code: https://github.com/ppriyank/RobustOnion
Area: Object Detection
Keywords: open-vocabulary object detection, robustness analysis, feature collapse, interpretability, low-quality images

TL;DR¶

Robust Onion peels open-vocabulary object detectors layer by layer using controlled synthetic degradations (turbulence, motion blur, pixelation), locating the failing layer through cross-severity feature overlap: robustness turns out to be governed almost entirely by vision-backbone architecture and depth (shallow layers fail first, while the enhancer and the language side contribute little). The findings are compiled into NN & TK0, a plug-and-play module with only 2.41M trainable parameters that matches end-to-end fine-tuning on real driving datasets with 96× fewer trainable parameters.

Background & Motivation¶

Vision-language-model-based open-vocabulary object detectors (OV-ODs such as GLIP, Grounding-DINO, OWL-ViT) are being deployed rapidly in security, medical imaging, environmental monitoring, and autonomous driving, because they accept new classes without fine-tuning. Yet they are also among the least transparent deep models: a complete OV-OD typically chains a multi-scale vision encoder, a text encoder, a feature enhancer (FPN or pixel decoder), a cross-modal fusion network, box predictors, and alignment losses, and a failure in any one of them shows up only as a handful of points of mAP.

The cleanest way to study how low-quality (LQ) noise actually hurts a model is to compare high-quality (HQ) and LQ images of the same scene against each other, but such "noise-paired" real-world datasets essentially do not exist — real adverse-weather datasets such as BDD-100K do not even contain the corresponding clean images. As a result, despite widespread deployment, the effect of visual distortions on OV-ODs is still reported only as "how many points did it drop": we know it broke, not where or why. Existing robustness work either targets CNN detectors such as Faster R-CNN, or, like LR0.FM, analyzes the much simpler task of VLM classification; none of it answers which part inside an OV-OD should be held responsible for robustness.

This paper's angle is to turn robustness from a black-box score into an attributable phenomenon. It first observes on real driving data that noise drives visual features into two distinguishable regimes — an observable collapse, where noisy features form clusters separate from clean features (e.g. cloudy, overcast), and a minimal collapse, where little or no separation is visible (e.g. snow, rain) — and then picks one synthetic proxy per regime (turbulence for observable, motion blur for minimal) to sidestep the missing paired data. With a controllable degradation source in hand, the OV-OD can be peeled like an onion: check whether each architectural component's features still overlap across noise severities, then check whether dataset composition and the language side participate at all. Core idea: decompose "robustness under noise" into an attributable causal chain — reproduce the feature collapse caused by real noise using synthetic degradations, use cross-severity feature overlap to find which layer fails first, and compile the conclusions directly into a lightweight adapter that only touches the shallow layers of the vision backbone.

Method¶

Overall Architecture¶

This is not a single-point method but an analysis protocol plus a module derived from its conclusions. The analysis covers six publicly available OV-ODs: RegionCLIP (including its ×4 variant), GLIP, FIBER, MM-Grounding-DINO (MM-GDINO), GLEE, and YOLO-World; the CNN-based ones (YOLO and un-fine-tuned RegionCLIP) are clearly weaker than the transformer-based ones, so the main analysis focuses on transformer detectors. Evaluation uses COCO val2017 (80 classes, mAP), LVIS miniVal (1,203 classes, AP), and ODinW-13 (13 datasets, APavg); the language side uses RefCOCO/+/g and Flickr30k (Recall@1); model-side conclusions are finally validated on real-world data including BDD-100K, DAWN, Foggy Cityscapes, Virtual KITTI 2, WiderFace, and VisDRONE.

One term must be pinned down first: in this paper "noise" means image-side degradation only (turbulence, motion blur, pixelation, applied to the input image alone), with text captions left untouched; class-name perturbations are out of scope, and the only text-side perturbation studied is the test-time prompt rewrite ablation (which turns out to be marginal). "Peeling the onion" is likewise not a literal layer-by-layer forward decomposition, but three orthogonal peel lines, each with its own controls: the vision stack (backbone layer position, enhancer, fusion network), the data side (dataset composition, object size/count/occlusion, annotation scheme), and the language side (text backbone, training-caption expressiveness, inference prompts). The criterion for peeling is feature overlap: a robust model should not distinguish noise severities, i.e. severity-5 noisy features should overlap severity-0 (clean) features in representation space; once a layer shows two separated clean/noisy clusters (feature collapse), that layer is the failure point. The overall procedure is a frozen model plus a set of controlled perturbations and grouping experiments rather than a serial multi-stage pipeline, so no framework diagram is drawn here.

Key Designs¶

1. Analysis protocol: reproduce the two real-noise feature-collapse regimes with synthetic degradations, and decouple absolute accuracy via relative robustness

Real HQ/LQ pairs are unavailable, and evaluating directly conflates "the model was always weak" with "the data got harder." The authors therefore start with a feature-level observation on real driving data and split noise-induced visual feature (variance) collapse into two regimes: observable collapse, where noisy features form clusters far from the clean ones (e.g. cloudy, overcast), and minimal collapse, where almost no separation is visible (e.g. snow, rain). They then pick one synthetic proxy per regime — turbulence (a pretrained GAN simulating hot-air phase distortion) for observable collapse, motion blur for minimal collapse — and use pixelation (downsample, then upsample back with bicubic interpolation) for severity sweeps. The crucial point is that the collapse regime is not an intrinsic property of a noise type but a function of intensity: raising the severity turns a minimal collapse into an observable one, which is what makes noise categories from a real dataset such as BDD-100K comparable on a single axis. Pixelation severity \(s\) is defined as \((H,W)\to(H/2^s, W/2^s)\to(H,W)\), and the authors emphasize that pixelation is not the same as low resolution (a \(256\times256\) image may still come from a downsampled high-resolution original) ⚠️ refer to the original paper.

For the metric, the paper deliberately avoids absolute accuracy and uses relative robustness throughout:

\[\text{Relative Robustness} = 1 - \frac{\text{Acc}_{\text{clean}} - \text{Acc}_{\text{noise}}}{\text{Acc}_{\text{clean}}} = \frac{\text{Acc}_{\text{noise}}}{\text{Acc}_{\text{clean}}}\]

It is the ratio of noisy to clean accuracy and is independent of a model's absolute level, so a ResNet detector and an EVA-02 detector can sit on the same ruler and COCO (mAP) can be compared with LVIS (AP). The cost is a boundary: at severities 4 and 5 the noisy accuracy approaches zero, relative robustness degenerates to the random level, and cross-model comparison stops being meaningful, so severity sweeps are only interpretable within a range. The authors also find that absolute accuracy and relative robustness are approximately linear in each other (Fig. 5 right), preserving the relative ranking of models, with the outliers mostly fine-tuned models — a linearity that is reused repeatedly later to explain prior observations.

2. Peeling the vision stack: locating the failing layer with cross-severity feature overlap

An OV-OD normally contains 3–5 transformers, and touching any one of them can affect robustness, which leaves practitioners with nowhere to start. The paper first compresses the search space by grouping detectors according to their vision backbone (ignoring enhancer, fusion, and text backbone differences) and finds that models sharing a backbone have highly consistent robustness. It then narrows the fine-grained analysis to the Swin-L backbone shared by GLIP-T, MM-GDINO-L, and GLEE-Pro — three architecturally different models whose maximum robustness gap across all noises is only 0.15 (COCO) / 0.08 (LVIS). Next it computes UMAP representations across severities for the last four backbone layers, the last enhancer layer, and the last fusion layer, comparing how much the severity-5 and severity-0 features overlap. Three conclusions follow: deeper backbone layers (#3, #4) overlap far more than shallow ones (#1, #2), so the shallow layers are where noise strikes first, and different models show similar collapse at the same depth, which directly explains why same-depth backbones have similar robustness (Swin-L and EVA-02 are both 24 blocks); the enhancer is essentially a simple convolutional block on top of the backbone, and its last-layer overlap is nearly identical to backbone layer #4, so it contributes almost nothing to robustness; and the fusion network, which cross-exchanges vision features at 192×192, 96×96, 48×48, and 24×24 with language, shows substantial cross-severity overlap, meaning cross-layer information exchange itself improves robustness — but the benefit comes from "cross-layer", not from "language".

Alongside this peel line, four correlation results support the verdict that the backbone dominates: robustness correlates positively with model size (Pearson 0.68/0.66 for pixelation, 0.78/0.72 for turbulence, 0.77/0.70 for motion blur on COCO/LVIS); it correlates only weakly with pretraining data size (0.35/0.23, 0.41/0.39, 0.42/0.22); fine-tuning is not a universal boost (clean COCO/LVIS fine-tuning clearly helps the ResNet-based RegionCLIP but barely affects the transformer-based FIBER-B); and the backbone depth ordering is ResNet < Swin-T (12 blocks, 27M) < Swin-B (12 blocks, 87M) ≤ Swin-L (24 blocks, 195M) ≃ EVA-02 (24 blocks, 303M) — beyond a certain scale, parameter count stops mattering and depth looks like the decisive factor.

3. Peeling the data and language sides: robustness is governed by the image domain, and neither annotations nor prompts can rescue it

Intuitively, more classes, more detailed descriptions, and prompts carrying degradation cues should all help robustness; the paper refutes this with three sets of controls. The most telling data-side result is COCO versus LVIS: they use exactly the same images while the annotation scheme changes from 80 classes to a 1,203-class long-tailed taxonomy, yet robustness is nearly identical — annotations are therefore not the governing variable. What actually makes a difference is the images themselves, because ODinW-13 holds robustness at ≈0.6, nearly twice that of COCO/LVIS. The difference lies in image composition: only about 10% of ODinW-13 objects are very small (≤32×32), versus about 42% for COCO and about 58% for LVIS; and about 50% of ODinW-13 images contain a single object, versus only about 12% for COCO and about 38% for LVIS. The matching fine-grained regularity is that objects of ≥96×96 are far more robust than those of ≤32×32; on single-object images nearly all detectors are extremely robust, robustness starts to drop once there are more than 3 objects, and saturates beyond about 10 (the jitter past 25 comes from too few samples in that bin). Occlusion between objects (the sum of pairwise IoU) barely affects robustness, a counter-intuitive result showing that the difficulty of a "crowded" image comes mainly from how many objects there are and how large they are, not from overlap.

The two language-side controls give the same negative answer. First, training-caption expressiveness: fine-tuning FIBER on RefCOCO (simple expressions), RefCOCO+ (appearance-based descriptions), and RefCOCOg (elaborate long sentences) respectively and then evaluating on COCO yields nearly identical robustness, with only a minor difference under turbulence. Second, inference-time prompts: captions from Flickr30k are rewritten by a Granite LLM into five prompts infused with a "low-resolution/pixelation" context (e.g. "a boy smiles in front of a stony wall in a city" becomes "in a low-resolution cityscape, a boy's smile is captured against a rough stone wall"), and evaluating the vision embedding fused with these five rewritten prompts leaves cross-severity robustness unchanged; on ODinW-13, fine-grained prompts versus superclass prompts differ by only ≈0.14 (pixelation) / 0.17 (turbulence) / 0.12 (motion blur). The conclusion is therefore that once visual features have been damaged, the language side cannot restore them — not because the text encoder is weak, but because it simply is not on the critical path.

4. NN & TK0: compiling the attribution results into a 2.41M-parameter plug-and-play module

The four analysis conclusions (backbone-dominated, shallow layers most fragile, cross-layer exchange effective, language side need not be trained) map directly onto two extremely lightweight components, both of which act only on the frozen OV-OD vision backbone and therefore do not damage the pretrained weights. The first is TK0, adapted from LR-TK0: the original method attaches trainable tokens on top of a frozen transformer and trains them by teacher-student distillation for low-resolution classification; here the expensive distillation is removed, only the trainable tokens are kept, and they are inserted into every backbone layer (shallow ones in particular) — a fixed 32×32 set of spatial tokens per layer, interpolated to that layer's spatial resolution and added to the frozen feature maps. Interpolation buys two things: LR-TK0 attaches a fixed number of prompts to every ViT layer and cannot handle varying \(H\times W\), whereas interpolation lets one token set adapt to every layer; and the overhead is lower, since for a 600×600 Swin-T input the 32×32 tokens add only 5.7% parameters, against 22.5% for the fixed-token design.

The second is NN, a trainable non-local block implemented as single-head self-attention whose query, key, and value are formed by concatenating the spatial tokens from all layers, letting tokens share spatial information across layers. It is in effect the minimal implementation of the design-2 finding that "the fusion network's benefit comes from cross-layer exchange" — note that NN has just 1 head and 1 layer, whereas the native fusion is a multi-layer multi-head transformer decoder. Combining both gives NN & TK0: 2.41M trainable parameters, 96× fewer than the 231.76M of end-to-end training, with the OV-OD itself left entirely frozen, so the loss in zero-shot COCO performance is correspondingly much smaller.

Loss & Training¶

GLIP-T is trained with its default configuration on the BDD-100K training set (698,630 image-text captions). The two baselines are E2E, which trains the whole GLIP end to end (including the text backbone), and Fuse, which freezes the model and trains only the fusion transformer — Fuse counts as a strong baseline because the fusion network is precisely the component that OV-ODs use by default to exchange information across backbone layers. Both baselines modify pretrained weights and therefore risk harming zero-shot ability. The paper's own components train far less: TK0 trains 1.57M parameters (147.4× lighter than E2E), NN trains 0.84M (275.7× fewer), and the combination 2.41M. In the supplementary experiments WiderFace is trained on Flickr30k and VisDRONE on UAVDT, with settings otherwise unchanged.

Key Experimental Results¶

Main Results¶

Table 1 reports zero-shot evaluation of BDD-100K-trained GLIP-T on real adverse-weather datasets (the parenthesized value is the change relative to the zero-shot baseline, with the sign indicating a gain or drop). DAWN contains categories such as partly cloudy (PC), snow (S), rain (R), fog (F), and overcast (O); Foggy Cityscapes uses fog intensity 0.02; Virtual KITTI 2 covers conditions such as City/F/O/R.

Method	Trainable params (M)	DAWN All	Foggy Cityscapes All	Virtual KITTI 2 All	COCO
Zero-shot	—	31.4	33.8	20.9	46.6
E2E	231.76	66.4 (+35.0)	36.0 (+2.2)	31.6 (+10.7)	3.0 (−43.6)
Fuse	91.85	65.1 (+33.7)	35.6 (+1.8)	29.8 (+8.9)	3.8 (−42.8)
TK0	1.57	45.1 (+13.7)	32.5 (−1.3)	26.1 (+5.2)	43.2 (−3.4)
NN	0.84	45.0 (+13.6)	33.4 (−0.4)	28.7 (+7.8)	22.1 (−24.5)
NN & TK0	2.41	55.2 (+23.8)	36.1 (+2.3)	29.9 (+9.0)	33.4 (−13.2)

A second table on cross-domain generalization makes the value of freezing the weights even clearer: on the Flickr30k→WiderFace and UAVDT→VisDRONE transfer chains, NN & TK0 with only 2.41M trainable parameters beats E2E on three of the four metric columns and beats the zero-shot baseline on all of them.

Method	Trainable params (M)	Flickr30k→WiderFace AP / AP50	UAVDT→VisDRONE AP / AP50
Zero-shot	—	11.98 / 26.49	20.44 / 27.09
E2E	231.76	10.86 / 24.69	18.68 / 30.56
Fuse	91.85	10.55 / 24.92	20.07 / 29.89
NN & TK0	2.41	13.23 / 28.71	19.34 / 30.97

Ablation Study¶

The paper's "ablation" is not module removal but component-wise peeling: for each component, ask how much it alone affects feature overlap under noise.

Peeled component	How it is observed	Observation	Contribution to robustness
Vision backbone (depth)	Grouping models by backbone	Models sharing a backbone have consistent robustness; ResNet < Swin-T < Swin-B ≤ Swin-L ≃ EVA-02	Dominant factor
Vision backbone (layer position)	Cross-severity UMAP overlap of backbone layers #1–#4	Shallow layers #1/#2 show clearly separated clean/noisy clusters; #3/#4 overlap strongly	Shallow layers are the failure source
Feature enhancer (FPN / pixel decoder)	UMAP of the last enhancer layer	Overlap comparable to backbone layer #4	Negligible
Fusion network	UMAP of the last fusion layer	Strong cross-severity overlap	Real, but from cross-layer exchange, not from language
Text backbone and language expressiveness	Text-backbone comparison, RefCOCO/+/g fine-tuning, LLM-rewritten prompts, fine-grained vs superclass prompts	All differences marginal (superclass prompts average Δ ≈0.14/0.17/0.12)	Negligible
Pretraining scale	Pretraining data size vs relative robustness	Pearson 0.35/0.23, 0.41/0.39, 0.42/0.22	Weak correlation
Fine-tuning	Clean COCO/LVIS fine-tuning	RegionCLIP (ResNet) benefits clearly, FIBER-B (transformer) barely changes	Not a universal boost
Annotation scheme	COCO vs LVIS (same images, different labels)	Robustness nearly equal	Negligible (image domain dominates)
Dataset composition	ODinW-13 vs COCO/LVIS	ODinW-13 ≈0.6, about twice the other two	A confounder: large isolated objects inflate the score

Key Findings¶

What fails first is the shallow layers of the vision backbone, not the text encoder. Noise splits clean and noisy features apart in the shallow layers, and every subsequent layer then works on a corrupted representation; the language side neither has the ability nor the channel to compensate, because fusion happens after the backbone. The design implication is direct: improving the language side is wasted effort compared with strengthening shallow layers or introducing cross-layer exchange.
ODinW-13 systematically overstates robustness. With about 50% single-object images and only about 10% very small objects, it sustains robustness scores close to twice those of COCO/LVIS; in the paper's illustration FIBER-B detects a handful of objects correctly on pixelated ODinW-13 even under heavy occlusion, but once the object count grows it merges everything into one large box. Reporting robustness on ODinW-13 alone therefore gives a false sense of safety.
Occlusion and robustness are nearly unrelated — counter-intuitive but self-consistent in the data. With object size and count controlled, the summed IoU between objects barely changes robustness, so the difficulty of a crowded image is dominated by object count and size. This explains why class frequency is almost uncorrelated with robustness (≈0.02/0.021) while mean class size shows a moderate correlation (0.52/0.45): the easiest COCO classes (parking meter, stop sign, toilet) are exactly those that tend to appear alone and at a regular size.
Freezing the pretrained weights pays off substantially. E2E and Fuse gain enormously on the real noisy sets (DAWN All +35.0 / +33.7) but collapse to 3.0 / 3.8 on original COCO (Δ −43.6 / −42.8), a textbook trade of generalization for robustness; NN & TK0 loses only 13.2 and still matches or slightly exceeds Fuse on DAWN and Foggy Cityscapes.
Neither NN nor TK0 suffices alone. TK0 alone is clearly below E2E/Fuse on real noise (DAWN All 45.1 vs 66.4), while NN alone drags COCO down to 22.1 (Δ −24.5). Adding tokens only in shallow layers and exchanging information across layers each supply half of the answer — the former adds representational capacity, the latter restores cross-layer information flow — and only together do they fully implement the analysis conclusions.

Highlights & Insights¶

Upgrading "robustness benchmarking" into "feature-collapse diagnosis." Cross-severity feature overlap is a probe that needs no labels, no retraining, and no paired data; as long as a model exposes intermediate layers it applies. The idea transfers to robustness attribution for any hierarchical model (segmentation, depth estimation, video understanding) without having to ablate modules one by one.
When substituting synthetic noise for unavailable HQ/LQ pairs, the paper states the correspondence explicitly (observable collapse ↔ turbulence, minimal collapse ↔ motion blur, tunable intensity ↔ pixelation). This turns an experiment that cannot be run into a controllable proxy experiment, and the authors acknowledge the correspondence is empirical — an unusual degree of restraint for this kind of analysis work.
Relative robustness decouples absolute accuracy, and the authors volunteer its weakness: it is approximately linear in clean accuracy, so "robust" is to a large extent a shadow of "accurate." That admission is what makes the later explanations of prior observations (e.g. GroundingDINO being more robust than OWL-ViT) convincing rather than selling correlation as causation.
NN & TK0 is the minimal realization of the analysis conclusions: if cross-layer exchange helps, shallow layers are fragile, and language need not be trained, then insert spatial tokens in shallow layers plus one single-head self-attention — 2.41M parameters, 96× cheaper, with the backbone fully frozen so zero-shot ability is preserved. This "conclusion → minimal verifiable module" loop is a rare landing for an analysis paper.

Limitations & Future Work¶

The noise is still synthetic. Turbulence, motion blur, and pixelation are only proxies for real degradations, and the authors concede that genuinely injecting degradation context would require a new dataset with noise-aware captions and annotations, left as future work — in other words, what this paper proves is that inference-time prompt engineering does not help; whether training-time injection does remains untested.
The metric breaks down at extreme severity. At severities 4 and 5 the noisy accuracy approaches zero and relative robustness degenerates to the random level, so cross-model comparison is meaningless there; although the paper flags this, some conclusions still rest on severity-5 feature visualizations.
Real data lacks a clean counterpart. BDD-100K has no clean images, so the collapse regimes observed on synthetic COCO can only be aligned to real noise categories indirectly; that alignment is indirect evidence, not a paired comparison of the same scene.
Limited coverage. The main analysis focuses on transformer detectors (CNN-based ones are excluded as unrepresentative), and the strong claim that "same depth implies similar collapse" rests mainly on two 24-block backbones, Swin-L and EVA-02 — a small sample.
Two observations of my own: NN used alone drops COCO from 46.6 to 22.1 (Δ −24.5), suggesting that exchanging information across layers without touching the shallow ones substantially harms generalization, a phenomenon the paper describes but does not explain; and TK0's 32×32 tokens and 5.7% overhead are measured for a 600×600 Swin-T input, leaving open whether they still hold for other backbones or input resolutions.

vs LR0.FM (Pathak et al., ICLR 2025): the direct predecessor on the same research line, also concerned with robustness under low-quality noise, but it analyzes VLM classification. This paper swaps in the far more complex OV-OD and extends LR-TK0 from ViT to hierarchical transformer detectors (dropping distillation and using interpolated tokens to fit each layer's resolution), generalizing the "shallow layers are fragile" conclusion from classification to detection.
vs Chhipa et al. (ECCV 2024): they observe that GroundingDINO is overall more robust than OWL-ViT; this paper does not treat that as a new architectural advantage but explains it as a direct consequence of the approximate linearity between robustness and clean accuracy — GroundingDINO simply has higher clean accuracy, so the observation reduces to a more general regularity.
vs COCO-O / Liu et al. (IJCV 2024): they find that DETR and Deformable DETR have similar robustness, violating the accuracy-robustness linearity; this paper explains it by the two models sharing the same vision backbone, so identical backbones produce identical collapse patterns and robustness is flattened.
vs DetCLIPv3 (Yao et al.): they report that swapping Swin-T for Swin-L improves robustness; this paper supplies the mechanism (backbone depth going from 12 to 24 blocks) and notes that simple Conv/MLP components such as the neck and the detection head have limited influence on feature collapse — OV-ODs already provide cross-layer exchange through their fusion design.
vs Zhou et al. (ICML 2022) and Bhojanapalli et al. (ICCV 2021): the former finds that shallow-layer features degrade more under noise, the latter argues that self-attention matters more than MLPs for robustness; this paper unifies both under one mechanism via UMAP — cross-position and cross-layer feature exchange reduces collapse, whereas pointwise Conv/MLP transformations do not change it.

Rating¶

Novelty: ⭐⭐⭐⭐ The first systematic interpretability study peeling apart the sources of robustness in open-vocabulary detectors; the "noise ≈ feature collapse" framing and the finding that the language side barely participates in visual robustness are both informative.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Six detectors, three detection benchmarks plus language benchmarks, six real-world datasets, three noise severities, feature-level UMAP diagnosis, and a point-by-point alignment with eight prior observations.
Writing Quality: ⭐⭐⭐⭐ The WHERE/WHY/WHAT/HOW organization is clear and the Takeaways are actionable, but the figures are dense, some small UMAP panels are hard to read, and the meaning of the Δ column in Table 1 requires the caption.
Value: ⭐⭐⭐⭐ The conclusions translate directly into design guidance (modify shallow vision-backbone layers plus cross-layer exchange) and are validated on real data by NN & TK0 with 96× fewer parameters; the main discount is that the conclusions rest on synthetic noise, leaving real HQ/LQ pairing an open gap.