FSDC-DETR: A Frequency-Spatial Domain Collaborative DETR for Small-Object Detection¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/nevereverinsomnia/FSDC-DETR
Area: Object Detection
Keywords: Small Object Detection, Detection Transformer, Frequency-Spatial Collaborative Modeling, Multi-Scale Feature Fusion, Discrete Wavelet Transform

TL;DR¶

To tackle high-frequency degradation and spectral aliasing caused by rigid spatial aggregation and downsampling in small object detection, FSDC-DETR introduces an explicit frequency-spatial collaborative framework combining dual-branch adaptive fusion, shunt cross-scale feature interaction, and dynamic wavelet downsampling, establishing new state-of-the-art results of 31.1 AP on VisDrone-DET2019 and 32.3 AP on AITODv2.

Background & Motivation¶

Small object detection (SOD) plays an essential role across real-world computer vision applications, including unmanned aerial vehicle (UAV) aerial surveillance, remote sensing, autonomous driving, and industrial defect detection. Unlike generic targets, small objects occupy extremely limited pixel footprints (e.g., in AITODv2, instances average only 12.7 pixels, and 86% are smaller than 16 pixels), suffering from low signal-to-noise ratios and vulnerability to cluttered backgrounds. While modern end-to-end Detection Transformers (such as RT-DETR, DEIM, and D-FINE) have eliminated hand-crafted anchors and NMS post-processing to deliver high inference efficiency and global semantic context, their direct application to small object detection remains severely constrained by feature representation attenuation.

The core bottleneck of existing detectors in SOD lies in the intrinsic spectral mismatch of their backbone representations and scale-transition mechanisms. Specifically, Vision Transformers (ViTs) like DINOv3 aggregate information via global self-attention, fundamentally functioning as low-pass filters that over-smooth fine-grained local boundaries. In contrast, CNN backbones like HGNetv2 naturally preserve high-frequency components but lack broad contextual modeling. Even when combined in dual-branch structures such as DEIMv2, the heterogeneous CNN and ViT features are simply concatenated and linearly projected, ignoring their distinct spectral characteristics and inducing spectral aliasing. Furthermore, conventional hybrid encoders rely on strided convolutions or average pooling during cross-scale aggregation and spatial reduction, progressively truncating and attenuating the subtle high-frequency cues and sharp textures indispensable for tiny targets.

Addressing this severe frequency aliasing and truncation requires abandoning implicit convolution smoothing in favor of an end-to-end framework that explicitly constructs, propagates, and preserves complementary frequency-spatial cues across all stages. Core idea: build a Frequency-Spatial Domain Collaborative DETR (FSDC-DETR) that couples dual-branch heterogeneous representations via frequency dynamic convolution and partial channel refinement, decouples cross-scale feature propagation into specialized spatial and frequency shunts, and preserves high-frequency structural integrity during scale transitions using learnable discrete wavelet decomposition and grouped convolution.

Method¶

Overall Architecture¶

FSDC-DETR is built upon the DEIMv2 architecture, comprising a dual-branch backbone (CNN + ViT), a Frequency-Spatial Collaborative Hybrid Encoder (FSC-Hybrid Encoder), and a Transformer decoder. The input image is fed into HGNetv2 (capturing local high-frequency details) and DINOv3 (extracting global low-pass semantics) simultaneously. Across multi-scale feature stages P2 through P5, the Dual-Branch Frequency-Spatial Adaptive Fusion (DBFSAF) module aligns the spectral disparities and refines discriminative cues with partial channel efficiency. The refined multi-scale features then enter the hybrid encoder, where Shunt Frequency-Spatial Feature Fusion (SFS-FF) handles bidirectional spatial and Fourier-transformed spectral interactions across feature levels. Across resolution transitions, the Frequency-Spatial Dynamic Downsampling (FSD-Down) operator replaces traditional strided convolutions to prevent spectral aliasing and retain high-frequency textures before features are dispatched to the Transformer decoder for bipartite matching and bounding box prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image (800×800)"] --> B1["HGNetv2 Branch<br/>Local High-Frequency Details"]
    A --> B2["DINOv3 Branch<br/>Global Low-Pass Semantics"]
    B1 --> C["Dual-Branch Frequency-Spatial Adaptive Fusion (DBFSAF)<br/>Frequency Dynamic Conv & Partial Channel Refinement"]
    B2 --> C
    C --> D["Frequency-Spatial Collaborative Hybrid Encoder (FSC-Hybrid Encoder)"]
    subgraph D ["FSC-Hybrid Encoder Pipeline"]
        direction TB
        D1["Shunt Frequency-Spatial Feature Fusion (SFS-FF)<br/>FFT Spectral Filtering & Spatial Structure Shunts"]
        D2["Frequency-Spatial Dynamic Downsampling (FSD-Down)<br/>DWT Sub-band Decomposition & Grouped Conv"]
        D1 <--> D2
    end
    D --> E["DETR Decoder & Detection Head<br/>Bipartite Matching & Box Regression"]
    E --> F["Output Detections (BBoxes & Classes)"]

Key Designs¶

1. Dual-Branch Frequency-Spatial Adaptive Fusion (DBFSAF): Spectral Diversification and Partial Channel Refinement

To resolve the frequency aliasing and truncation that arise when naively concatenating heterogeneous CNN and ViT feature maps, DBFSAF introduces Frequency Dynamic Convolution (FDConv) paired with a Partial Frequency-Spatial Refinement (PFSR) mechanism. For feature maps \(F_L^P\) (CNN) and \(F_H^P\) (ViT) at stage \(P \in \{2, 3, 4, 5\}\), the concatenated features are first processed by FDConv, which dynamically adjusts filter weights across different spatial locations and frequency bands, assigning selective frequency emphasis to object boundaries versus smooth backgrounds. To prevent redundant computation across all channels, a partition ratio \(\gamma\) (set to 0.5) divides the \(C\)-channel output into an identity shortcut path \(F_C^P\) and an interactive refinement path \(F_{FS}^P\):

\[\hat{F}_{\mathbf{C}}^P, F_{\mathbf{FS}}^P = \text{Partial}_\gamma\left(\text{FDConv}\left(\text{Concat}[F_L^P, F_H^P]\right)\right)\]

The interactive path \(F_{FS}^P\) is then concurrently processed by a Multi-Kernel Spatial Refine Unit (MKSRU) and a Frequency Processing Unit (FPU). MKSRU deploys multi-dilation depthwise convolutions with dilation rates 1, 2, and 3 across \(5\times 5\) and \(7\times 7\) kernels to capture multi-scale spatial structures, while FPU extracts frequency-sensitive responses. The two outputs are modulated via channel attention and Hadamard product before concatenating back with \(F_C^P\), preserving critical high-frequency cues with minimal computational overhead.

2. Shunt Frequency-Spatial Feature Fusion (SFS-FF): Structure-Aware Cross-Scale Spectral Propagation

Conventional hybrid encoders rely on standard convolutions to merge multi-scale feature maps, which blurs fine-grained details during inter-scale aggregation. SFS-FF addresses this by decoupling multi-scale features into three dedicated shunts using a \(1\times 1\) convolution: a spatial refinement path \(F_S\) (allocated \(C/2\) channels), a frequency refinement path \(F_F\) (allocated \(3C/8\) channels), and a residual connection path \(\acute{F}\) (allocated \(C/8\) channels).

The spatial path \(F_S\) is processed by a Spatial Refine Module (SRM) to consolidate geometric structures, whereas the frequency path \(F_F\) is converted into the frequency domain via a 2D Fast Fourier Transform (FFT). Within the frequency domain, a \(3\times 3\) convolution performs spectral filtering and adaptive reweighting, followed by an Inverse Fast Fourier Transform (IFFT) returning the features to the spatial domain:

\[F_{\mathbf{out}} = \text{Conv}_{1\times 1}\left(\text{Concat}\left[\text{IFFT}(\text{Conv}_{3\times 3}(\text{FFT}(F_S))), \text{SRM}(F_F), \acute{F}\right]\right)\]

By segregating spatial geometry from Fourier spectral filtering and recombining them collaboratively, SFS-FF ensures that small objects maintain high representation saliency across deeper encoder layers.

3. Frequency-Spatial Dynamic Downsampling (FSD-Down): Discrete Wavelet Decomposition and Grouped Convolution

Standard downsampling operations with stride 2 suffer from aliasing artifacts, and average pooling acts as an aggressive low-pass filter that strips high-frequency signals before frequency modeling can occur. FSD-Down resolves this by marrying 2D Discrete Wavelet Transform (DWT) with grouped convolution (GConv).

The module utilizes a Haar 2D-DWT to decompose the input feature map \(F_{\text{in}}\) into four lossless half-resolution \((H/2 \times W/2)\) sub-bands: low-frequency approximation \(F_{LL}\), along with horizontal \(F_{LH}\), vertical \(F_{HL}\), and diagonal \(F_{HH}\) high-frequency details. In parallel, a depthwise grouped convolution with kernel size and stride 2 maintains continuous spatial geometry. A cross-domain channel attention vector \(F_{\mathbf{CA}}\) is then derived via Global Average Pooling (GAP) and Sigmoid activation, combined with a learnable scaling factor \(\rho\) to adaptively modulate sub-band responses:

\[F_{\mathbf{CA}} = \sigma\left(\text{GAP}\left(\text{Concat}\left[\text{DWT}(F_{\mathbf{in}}), \text{Conv}_{1\times 1}(\text{GConv}(F_{\mathbf{in}}))\right]\right)\right)\]

This dynamic sub-band calibration allows the detector to emphasize high-contrast edges and corner frequencies without sacrificing spatial consistency, completely preventing frequency degradation during downsampling.

Loss & Training¶

FSDC-DETR follows DEIMv2 end-to-end training objectives comprising classification and bounding box regression losses. The model is trained on 8 NVIDIA H200 GPUs for 100 epochs with a total batch size of 64 and an input resolution of \(800 \times 800\). Optimization uses AdamW with an initial learning rate of \(5 \times 10^{-4}\), momentum of 0.9, and weight decay of \(1.25 \times 10^{-4}\). Data augmentations include ColorJitter, Zoom-out, Mosaic, MixUp, and CopyBlend. Dense one-to-one (dense O2O) matching is engaged during the first 50% of epochs to accelerate bipartite convergence and disabled thereafter. In the final two epochs, all data augmentations are turned off to align with the real data distribution.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on the official test splits of VisDrone-DET2019 and AITODv2 at an input resolution of \(800 \times 800\), benchmarking against state-of-the-art real-time detectors and DETR models:

Dataset	Model	Params (M)	AP	AP50	AP75	APS	APM	APL
VisDrone-DET2019	YOLOv11-L	25.3	22.6	37.3	23.5	12.5	33.2	47.8
VisDrone-DET2019	D-FINE-L	31.0	26.1	45.6	26.4	15.8	37.5	53.6
VisDrone-DET2019	DEIMv2-L (Baseline)	32.0	24.7	43.3	24.6	14.2	35.4	52.7
VisDrone-DET2019	RT-DETRv4-L	31.0	26.4	45.7	26.7	16.1	37.3	49.3
VisDrone-DET2019	FSDC-DETR (Ours)	40.3	31.1	52.3	31.9	21.0	41.8	58.1
AITODv2	YOLOv11-L	25.3	17.6	35.9	15.1	16.6	33.0	-
AITODv2	D-FINE-L	31.0	25.1	56.2	18.8	23.9	43.2	-
AITODv2	DEIMv2-L (Baseline)	32.0	25.7	55.7	20.6	24.4	43.7	-
AITODv2	RT-DETRv4-L	31.0	26.8	58.3	20.6	23.0	44.5	-
AITODv2	FSDC-DETR (Ours)	40.3	32.3	63.9	28.7	31.3	48.5	-

Ablation Study¶

A step-by-step ablation study on the VisDrone-DET2019 test split investigates the individual and combined contributions of DBFSAF, SFS-FF, and FSD-Down relative to the DEIMv2-L baseline:

Config	DBFSAF	SFS-FF	FSD-Down	AP	AP50	AP75	APS	APM	APL	Note
Baseline (DEIMv2-L)	✗	✗	✗	24.7	43.3	24.6	14.2	35.4	52.7	Naive concat & conv downsampling
+ DBFSAF	✓	✗	✗	29.7	49.7	30.5	19.8	40.6	55.5	Substantial +5.0 AP gain
+ SFS-FF	✗	✓	✗	25.9	44.9	26.0	15.5	36.8	53.7	Shunt cross-scale frequency interaction
+ FSD-Down	✗	✗	✓	25.4	44.2	25.6	15.1	36.4	52.9	Wavelet downsampling alone
Combination A	✗	✓	✓	26.3	45.5	26.4	16.4	37.4	54.5	SFS-FF + FSD-Down interaction
Combination B	✓	✗	✓	30.0	50.5	30.6	19.8	40.8	55.7	DBFSAF + FSD-Down
Combination C	✓	✓	✗	30.3	50.9	30.8	20.0	41.3	56.0	DBFSAF + SFS-FF
Full Model (FSDC-DETR)	✓	✓	✓	31.1	52.3	31.9	21.0	41.8	58.1	Optimal synergy across all 3 modules

Furthermore, hyperparameter ablation on the channel partition ratio \(\gamma\) in DBFSAF demonstrates that \(\gamma=0\) yields 29.7 AP, climbing to 30.8 AP at \(\gamma=0.375\), peaking at 31.1 AP (and 21.0 APS) at \(\gamma=0.5\), and mildly degrading to 30.9 AP (\(\gamma=0.625\)) and 30.0 AP (\(\gamma=1.0\)), proving that partial channel refinement avoids redundant interference while boosting accuracy.

Key Findings¶

Dual-branch frequency alignment is the dominant performance driver: Adding DBFSAF alone boosts AP by 5.0 points (24.7 \(\to\) 29.7) and small-object APS by 5.6 points (14.2 \(\to\) 19.8), demonstrating that resolving spectral aliasing between CNN and ViT is the primary bottleneck in dual-branch detection architectures.
Wavelet downsampling dramatically improves strict localization precision: Strict localization metric AP75 rises from 24.6 to 31.9 (+7.3) in the full model. The preservation of horizontal, vertical, and diagonal sub-band details by DWT provides the spatial-frequency sharpness needed for accurate small bounding box regression.
Benefits extend robustly across all object scales: Beyond small targets, medium objects gain +6.4 APM (35.4 \(\to\) 41.8) and large objects gain +5.4 APL (52.7 \(\to\) 58.1), verifying that explicit frequency-spatial interaction enhances the overall signal-to-noise ratio of multi-scale representations.

Highlights & Insights¶

Closed-loop frequency-spatial pipeline: Rather than treating frequency transformations as isolated plug-ins, FSDC-DETR establishes a systematic cycle: backbone alignment (DBFSAF), cross-scale propagation (SFS-FF), and scale-transition preservation (FSD-Down). This end-to-end continuity stops high-frequency degradation across feature hierarchies.
Efficient partial channel refinement: DBFSAF applies intensive multi-kernel spatial and frequency operations to only 50% of the channel capacity, keeping the remaining channels intact. This design strikes an outstanding balance between discriminative representation learning and computational throughput.
Drop-in lossless downsampling operator: FSD-Down replaces heuristic strided convolutions and pooling layers with parameterized Haar wavelets and grouped convolutions. This design can be readily transferred to broader vision models, such as YOLO or semantic segmentation backbones, to prevent detail loss during spatial reduction.

Limitations & Future Work¶

Increased parameter count and memory bandwidth: The introduction of DWT/FFT operations and frequency dynamic convolution raises model parameters from 32.0M to 40.3M, which may impose heavier memory bandwidth burdens on compute-constrained edge UAV hardware.
Sensitivity to heavy motion blur: High-frequency spectral modules rely on distinct edge gradients; in aerial capture scenarios with severe camera shaking or out-of-focus blur, high-frequency signals degrade into noise, potentially diminishing the gains of spectral filtering.
Future directions: Exploring model compression, low-rank frequency kernel factorization, and injecting frequency-aware guidance directly into the cross-attention layers of the DETR decoder to guide target query localization.

vs DEIMv2 [24]: While DEIMv2 pioneered the dual-branch DINOv3 + CNN hybrid architecture, it relies on simple concatenation and standard downsampling. FSDC-DETR identifies the underlying spectral mismatch between low-pass ViTs and high-pass CNNs, introducing DBFSAF, SFS-FF, and FSD-Down to achieve a +6.4 AP leap on VisDrone.
vs RT-DETRv4 [35]: RT-DETRv4 utilizes DINOv3 high-level representations to supervise the AIFI encoder via knowledge distillation. FSDC-DETR directly models frequency features in the primary feature flow, outperforming RT-DETRv4 by 4.9 APS on VisDrone and showing marked superiority on tiny targets in AITODv2.
vs UAV-DETR [87]: Although UAV-DETR attempted frequency-aware downsampling, it combined average pooling with FFT, causing average pooling to act as a low-pass filter that pre-emptively destroyed high frequencies. FSDC-DETR adopts DWT and grouped convolutions to preserve complete sub-band details without pre-filtering truncation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pioneers a comprehensive frequency-spatial collaborative DETR architecture with elegant wavelet downsampling and partial channel refinement.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks and detailed component ablations on VisDrone-DET2019 and AITODv2, showcasing extraordinary gains on small objects.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, clear mathematical formulations, and insightful analysis of spectral characteristics.
Value: ⭐⭐⭐⭐⭐ Provides a powerful reference framework for tiny object detection in remote sensing, aerial robotics, and industrial inspection.