Explicit Semantic–Spatial Alignment for Open-Vocabulary Object Detection¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/2014.pdf
Area: Object Detection
Keywords: Open-Vocabulary Object Detection, Semantic-Spatial Alignment, Frequency-Aware Feature Fusion, Vision-Language Models, DINOv2

TL;DR¶

To tackle the semantic-spatial discrepancy and resulting localization degradation in open-vocabulary object detection, ESSA-OVD builds a dual-stream architecture combining frozen CLIP and DINOv2 backbones, progressively injecting high-frequency spatial cues via a Spatial Adapter and Frequency-Aware Fusion, while preserving zero-shot semantic matching through a Pure Dense routing strategy with only 3M extra trainable parameters.

Background & Motivation¶

Open-Vocabulary Object Detection (OVD) aims to break the closed-world assumption by localizing and categorizing novel objects specified by arbitrary natural language text prompts. Powered by large-scale contrastive Vision-Language Models (VLMs) such as CLIP, existing OVD systems project visual regions and text queries into a shared cross-modal latent space. However, adapting image-level contrastive VLMs to dense, region-level detection exposes a fundamental tension termed the Semantic-Spatial Discrepancy. Contrastive pretraining intrinsically behaves as a low-pass filter, aggregating global semantic identity while suppressing fine-grained spatial and structural variations to achieve spatial invariance. Consequently, visual features extracted by CLIP exhibit diffuse spatial activations with blurry object boundaries, leading to severe bounding box regression drift and localization instability.

In sharp contrast, visual foundation models trained with self-supervised patch-level objectives (notably DINOv2) retain sharp boundaries and fine-grained spatial structures, yet they lack language alignment for zero-shot text classification. Prior attempts to reconcile this gap primarily rely on external supervision tricks—such as knowledge distillation from CLIP teachers, self-training with pseudo-labels, or region-level prompt tuning. While helpful, these techniques fail to resolve the core spectral deficiency inherent in CLIP-like representations, where the high-frequency spectra essential for delineating precise object boundaries are systematically missing. Furthermore, naively fusing DINOv2 features into the detection backbone via late concatenation or channel addition introduces distribution mismatches and triggers severe semantic drift, which impairs CLIP's pristine zero-shot classification space.

To overcome this dilemma, the model must explicitly recover the missing spatial high frequencies while preserving the integrity of the language-aligned semantic manifold. Core idea: build a dual-stream architecture with frozen CLIP and DINOv2 backbones, standardize the auxiliary spatial manifold via a lightweight Spatial Adapter, selectively inject high-frequency spatial residuals through a norm-preserving Frequency-Aware Fusion module, and physically decouple feature routing via a Pure Dense strategy so that classification features remain entirely pristine.

Method¶

Overall Architecture¶

ESSA-OVD establishes a staged, dual-stream semantic-spatial alignment framework. The pipeline comprises a frozen CLIP ViT backbone for the semantic stream, a frozen DINOv2 ViT backbone for the auxiliary spatial stream, a lightweight Spatial Adapter, a Frequency-Aware Fusion (FAF) module, and a Pure Dense routing mechanism feeding into a Deformable DETR detector head. Given an input image, both frozen backbones extract multi-level token representations. The raw spatial features are first projected and standardized by the Spatial Adapter to match the semantic channel space and eliminate covariate shift. Next, the FAF module isolates high-frequency spatial details via local smoothing subtraction and injects them into multi-level encoder features with norm-preserving rescaling (IsoNorm). Finally, the Pure Dense strategy routes the high-frequency enhanced features exclusively to the Transformer encoder for localization, while preserving the uncorrupted semantic features for the dense classification branch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B1["Frozen Semantic Stream<br/>CLIP ViT Backbone"]
    A --> B2["Frozen Spatial Stream<br/>DINOv2 ViT Backbone"]
    B2 --> C["Spatial Adapter<br/>1x1 Conv + GroupNorm Manifold Alignment"]
    B1 --> D["Frequency-Aware Fusion<br/>High-Pass Residual Extraction + IsoNorm Injection"]
    C --> D
    D --> E["Pure Dense Strategy<br/>Encoder Layers Injected / Classification Kept Pristine"]
    E --> F1["Deformable DETR Encoder<br/>Multi-Level High-Frequency Localization"]
    E --> F2["Classification Branch<br/>Pristine Text-Image Similarity Matching"]
    F1 --> G["Final Open-Vocabulary Predictions"]
    F2 --> G

Key Designs¶

1. Spatial Adapter: Heterogeneous Manifold Rectification and Statistical Normalization Raw DINOv2 features reside on a latent manifold optimized for self-supervised instance discrimination, which has substantial statistical differences and channel dimension discrepancies compared to CLIP ($C_{spa} = 1024$ vs. $C_{sem} = 768$). Directly injecting these raw features causes severe covariate shift and disrupts semantic stability. The Spatial Adapter $\mathcal{A}(\cdot)$ maps the $l$-th layer spatial feature $F^{spa}_l \in \mathbb{R}^{C_{in} \times H_l \times W_l}$ into the semantic feature space: $$\hat{F}^{spa}_l = \mathcal{A}(F^{spa}_l) = \sigma\left(\text{GN}_G\left(\mathbf{W}_{proj} F^{spa}_l\right)\right)$$ where $\mathbf{W}_{proj}$ is a $1 \times 1$ convolution compressing channels from $C_{in}$ to $C_{out}$, $\sigma$ denotes ReLU activation, and $\text{GN}_G$ represents Group Normalization with $G=32$ groups. Group Normalization standardizes feature statistics to zero mean and unit variance independently of batch size, suppressing background clutter and ensuring auxiliary spatial signals do not overwhelm primary semantic representations.

2. Frequency-Aware Fusion: Spectral Decomposition and Norm-Preserving Injection Even after manifold projection, full-spectrum DINOv2 representations carry low-frequency layout and textural patterns that can conflict with CLIP's high-level semantics. Drawing inspiration from unsharp masking, the FAF module decomposes the adapted spatial feature into a smoothed low-frequency component using average pooling $\mathcal{P}_{low}$ with kernel size $k=3$, and extracts the high-frequency residual $\mathbf{H}_l$: $$\mathbf{H}_l = \hat{F}^{spa}_l - \mathcal{P}_{low}(\hat{F}^{spa}_l)$$ This high-pass operation isolates object contours and boundary discontinuities while filtering out smooth background signals. The structural residual is then projected by linear layer $\mathcal{W}_{align}$ and gated by a learnable scalar $\alpha$ (initialized to zero for conservative training): $\tilde{F}^{sem}_l = F^{sem}_l + \alpha \cdot \mathcal{W}_{align}(\mathbf{H}_l)$. To counteract semantic drift caused by feature norm inflation, the module applies IsoNorm: $$\text{IsoNorm}(\tilde{F}^{sem}_l, F^{sem}_l) = \frac{\tilde{F}^{sem}_l}{\|\tilde{F}^{sem}_l\| + \epsilon} \cdot \|F^{sem}_l\|$$ IsoNorm decouples directional updates from feature scale and rescales the fused feature back to the original CLIP norm, preventing downstream Transformer layers from suffering numerical distortion.

3. Pure Dense Strategy: Decoupling Localization Routing from Open-Vocabulary Classification In the Deformable DETR framework, multi-level feature representations must serve two fundamentally opposing roles. The multi-level encoder features $\{F_l\}_{l=1}^3$ require fine-grained spatial gradients and sharp boundaries for proposal generation and bounding box regression; conversely, the dense feature map $F_{cls}$ is utilized for computing cosine similarity against offline CLIP text embeddings, demanding maximum semantic invariance and zero domain perturbation. Fusing spatial cues into $F_{cls}$ inevitably introduces subtle semantic degradation. The Pure Dense strategy enforces an explicit routing bifurcation: $$F^{out}_l = \Phi_{FAF}(F^{sem}_l, F^{spa}_l), \quad \forall l \in \{1, 2, 3\}; \qquad F^{out}_{cls} = F^{sem}_{cls}$$ By bypassing spatial fusion entirely for $F_{cls}$, the open-vocabulary classification space remains strictly pristine while the regression branch reaps the full benefits of high-frequency spatial grounding.

Loss & Training¶

The overall detector is optimized using the bipartite Hungarian matching algorithm, balancing classification and box regression objectives: $$\mathcal{L}_{total} = \lambda_{cls} \mathcal{L}_{cls} + \lambda_{L1} \mathcal{L}_{L1} + \lambda_{giou} \mathcal{L}_{giou}$$ with loss coefficients set to $\lambda_{cls} = 2.0$, $\lambda_{L1} = 5.0$, and $\lambda_{giou} = 2.0$. The classification loss $\mathcal{L}_{cls}$ adopts Focal Loss applied to softmax-normalized cosine similarities between decoder object query embeddings and offline base category text embeddings generated with an ensemble of 80 prompt templates. The regression loss combines normalized coordinate $\ell_1$ distance and scale-invariant GIoU loss. Both CLIP and DINOv2 backbones remain completely frozen during training, introducing only approximately 3M trainable parameters in the adapter, FAF modules, and detector head. The network is trained with the AdamW optimizer at a learning rate of $1 \times 10^{-4}$ and weight decay of $1 \times 10^{-4}$, stabilized by Exponential Moving Average (EMA) with a decay of 0.999.

Key Experimental Results¶

Main Results¶

ESSA-OVD is extensively evaluated on the OV-COCO benchmark (48 base / 17 novel classes, reporting novel class $AP_{50}^{Novel}$) and the long-tailed OV-LVIS benchmark (reporting rare class $mAP_r$). The method sets new state-of-the-art records across multiple backbone capacities:

Dataset	Backbone Config	Supervision	Ours	Prev. SOTA	Gain
OV-COCO	CLIP ViT-L/14 + DINOv2	CLIP	47.6 $AP_{50}^{Novel}$	CCKT-Det++ (46.0, Swin-B)	+1.6 $AP_{50}$
OV-COCO	CLIP ViT-L/14 + DINOv2	CLIP	47.6 $AP_{50}^{Novel}$	CLIPSelf (44.3, ViT-L/14)	+3.3 $AP_{50}$
OV-COCO	CLIP ViT-B/16 + DINOv2	CLIP	40.0 $AP_{50}^{Novel}$	CLIPSelf (37.6, ViT-B/16)	+2.4 $AP_{50}$
OV-LVIS	CLIP ViT-L/14 + DINOv2	CLIP	41.3 $mAP_r$	OV-DQUO (39.3, ViT-L/14)	+2.0 $mAP_r$
OV-LVIS	CLIP ViT-L/14 + DINOv2	CLIP	41.3 $mAP_r$	CLIPSelf (34.9, ViT-L/14)	+6.4 $mAP_r$
OV-LVIS	CLIP ViT-L/14 + DINOv2	CLIP	41.3 $mAP_r$	CoDet (37.0, ViT-L/14, Caption)	+4.3 $mAP_r$
OV-LVIS	CLIP ViT-L/14 + DINOv2	CLIP	41.3 $mAP_r$	RO-ViT (34.1, ViT-H/16)	+7.2 $mAP_r$
OV-LVIS	CLIP ViT-B/16 + DINOv2	CLIP	32.8 $mAP_r$	OV-DQUO (29.7, ViT-B/16)	+3.1 $mAP_r$

In zero-shot cross-dataset evaluation without any fine-tuning (models trained on OV-LVIS and evaluated on Objects365 and COCO), ESSA-OVD achieves 17.1 AP on the challenging Objects365 dataset (365 classes), outperforming previous best CCKT-Det++ (15.2 AP) by 1.9 AP, confirming robust domain transferability.

Ablation Study¶

Component-wise ablations on the OV-LVIS benchmark using the ViT-L/14 backbone reveal the distinct impact of each proposed mechanism:

Config #	Semantic-Spatial Alignment (DINOv2)	Spatial Adapter	Frequency-Aware Fusion	Rare Class $mAP_r$	Note
1	✗	✗	✗	38.1	Baseline CLIP-only detector
2	✓	✗	✗	39.6	Direct feature introduction (+1.5)
3	✓	✓	✗	40.3	Manifold alignment via Spatial Adapter (+0.7)
4	✓	✗	✓	40.5	FAF without proper manifold alignment (+0.9)
5 (Full)	✓	✓	✓	41.3	Full staged alignment pipeline (+3.2 total)

Further design explorations validate specific component architectural choices: - Adapter Type Comparison: No adapter (40.5 $mAP_r$), LFA Adapter (40.6 $mAP_r$), CBAM Adapter (41.0 $mAP_r$), proposed lightweight Spatial Adapter (41.3 $mAP_r$). The proposed adapter achieves the highest accuracy while drastically reducing computational overhead. - Fusion Paradigm Comparison: Baseline without fusion (40.3 $mAP_r$), Channel Fusion (40.1 $mAP_r$, performance degrades), Gated Spatial Fusion (40.4 $mAP_r$), proposed Frequency-Aware Fusion (41.3 $mAP_r$). Dense channel attention causes mutual interference across multimodal channels, whereas spectral separation delivers clean boundary priors.

Key Findings¶

Staged alignment is strictly necessary: Injecting frequency features without the Spatial Adapter yields only 40.5 $mAP_r$, noticeably lagging behind the full model at 41.3 $mAP_r$. Features from distinct self-supervised paradigms must be normalized in statistical distribution before spectral filtering.
High-frequency decomposition avoids channel dilution: Standard channel attention fusion actually degrades performance to 40.1 $mAP_r$ (below the 40.3 $mAP_r$ unfused baseline), proving that blindly correlating large heterogeneous channel spaces introduces severe noise. In contrast, high-pass spatial filtering provides precise geometric boundaries without confusing semantic tokens.
Asymmetric backbone scaling maintains efficiency: When replacing the spatial stream backbone from DINOv2 ViT-L/14 to the much smaller ViT-B/14, rare class performance drops by only 0.5 $mAP_r$ (41.3 to 40.8), demonstrating that the performance gain stems from structural complementarity rather than brute-force parameter expansion.

Highlights & Insights¶

Root-cause spectral perspective on VLM localization: The paper pinpoints that contrastive vision-language pretraining acts as a low-pass filter suppressing boundary high frequencies, shifting the OVD paradigm from complex external supervision to explicit spectral compensation.
Pure Dense routing decoupling: Elegant separation of feature streams ensures that the high-frequency structural cues enhance regression in the Transformer encoder while the dense classification space preserves pristine zero-shot language alignment.
Magnitude preservation with IsoNorm: By combining zero-initialized gating with norm-preserving rescaling, the framework circumvents feature norm distortion and prevents semantic drift.

Limitations & Future Work¶

Dual-stream inference latency: Maintaining both CLIP and DINOv2 backbones simultaneously increases memory footprint and computational overhead during deployment.
Fixed spatial filter kernel: The unsharp masking relies on a heuristic $3 \times 3$ average pooling kernel without dynamic adaptation across object scales or multi-frequency bands.
Future directions: Investigating offline knowledge distillation to compress DINOv2 high-frequency priors directly into a single-stream detector backbone for real-time edge deployment.

vs ViLD / CCKT-Det++: Distillation-based methods transfer semantic embeddings across student and teacher networks but remain constrained by single-stream CLIP feature blurriness. ESSA-OVD explicitly introduces DINOv2 spatial priors, outperforming CCKT-Det++ (46.0) with 47.6 $AP_{50}^{Novel}$ on OV-COCO.
vs CORA / DetPro: Prompt tuning methods guide attention weights through learnable prompts but cannot recover missing high frequencies from frozen weights. ESSA-OVD remedies the spectral deficit directly at the feature level.
vs Coarse VFM / SAM Integration: Existing methods that integrate SAM or DINO often use naive concatenation or late proposal refinement, leading to semantic drift. ESSA-OVD's frequency decomposition and Pure Dense strategy guarantee that open-vocabulary semantic alignment is preserved without distortion.

Rating¶

Novelty: ⭐⭐⭐⭐ [Insightful spectral decomposition perspective combined with clean manifold normalization and Pure Dense routing]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across OV-COCO, OV-LVIS, zero-shot Objects365 transfer, and detailed ablation breakdowns]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured empirical validation]
Value: ⭐⭐⭐⭐ [Provides a practical, highly reproducible framework for multimodal foundation model feature fusion in dense perception tasks]