Adaptive Spectrum-Aware Feature Disentangled Network for Small Object Detection¶

Conference: ECCV2026
arXiv: 2606.29029
Code: https://github.com/ManOfStory/SFDNet
Area: Object Detection
Keywords: Small Object Detection, Spectrum Disentanglement, State Space Models, Prototype Distillation, Background Suppression

TL;DR¶

SFDNet explicitly disentangles features into low, middle, and high-frequency components in the spectral domain. It designs specialized Mamba scanning strategies for each spectrum to perform context modeling, followed by adaptive fusion. Meanwhile, it constructs class-level prototypes to compress the feature distribution of intra-class objects via contrastive distillation, significantly outperforming previous SOTA methods on AI-TOD, SODA-D, and SODA-A.

Background & Motivation¶

Small Object Detection (SOD) is crucial in applications such as remote sensing, UAV inspection, and autonomous driving. Its fundamental difficulty lies in the fact that small targets only occupy dozens or even a few pixels, lacking sufficient visual cues and easily blending with the background. In recent years, the industry has advanced from three main directions: data augmentation (copy-paste, GAN/diffusion model generation), label assignment (Wasserstein distance-based adaptive strategies), and feature enhancement. Among these, feature enhancement is the most core component: the high-level semantics in the FPN pyramid have severely decayed, and the effective signals of small targets are almost submerged in background noise. Latest methods like HS-FPN attempt to suppress low-frequency background components using high-pass filters, while SET highlights frequency bands related to small objects via spectral enhancement. However, they are all based on an implicit assumption—that noise is concentrated in specific frequency bands.

In reality, this is not true: empirical measurements show that different frequency components have vastly different sensitivities to targets of various scales—low frequencies respond strongest to background structures and large-scale interference; middle frequencies can cover objects ranging from extremely tiny to medium scales; high frequencies are particularly sensitive to extremely tiny targets. Noise is distributed across the entire spectrum, so processing only a single frequency band inevitably loses complementary information from other bands.

The second key issue is that the semantic correlation between objects of the same category is not fully utilized. Small objects of the same category often share similar appearance patterns (such as rows of vehicles or roofs in remote sensing images). However, existing methods either enhance representations for each instance independently via super-resolution, feature imitation, or memory retrieval, or perform instance-level relationship grouping using fixed metrics like IoU or cosine similarity. The former only focuses on individuals and ignores category commonalities, while the latter has limited generalization. What is truly needed is class-level semantic supervision: a mechanism that can explicitly model the "shared features of all objects belonging to that category," forcing the features of intra-class small objects to compress together in the embedding space.

To solve these two issues, SFDNet proposes two complementary technical paths working in synergy. Core Idea: On one hand, Adaptive Spectrum Disentanglement (ASD) is used to explicitly decompose FPN features into low, middle, and high-frequency components, using specialized Mamba scanning strategies for context modeling in each component before fusing them with learnable weights to achieve background disambiguation across the entire spectrum. On the other hand, Class Prototype Distillation (CPD) constructs and maintains class-level prototype embeddings, utilizing contrastive learning to pull proposal features closer to their corresponding class prototypes. ASD ensures that each object isolates clean spectral features from noise, while CPD ensures that objects of the same category are tightly compressed in these feature spaces.

Method¶

Overall Architecture¶

SFDNet is based on the standard "Backbone + FPN + Detection Head" paradigm. Its core lies in introducing the Adaptive Spectrum Disentanglement (ASD) module after the FPN to replace conventional convolution/attention enhancement, and appending a Class Prototype Distillation (CPD) branch during training to provide contrastive supervision. The overall pipeline is: the input image passes through the Backbone to extract multi-scale features \(\{C_2, C_3, C_4, C_5\}\), which are aggregated top-down by the FPN into \(\{P_2, P_3, P_4, P_5\}\). Taking an arbitrary pyramid layer \(P\) as an example, the ASD module first disentangles \(P\) into three complementary components—low-frequency \(P_{\text{low}}\), middle-frequency \(P_{\text{mid}}\), and high-frequency \(P_{\text{high}}\)—via Difference of Gaussian (DoG). Each component is transformed into the frequency domain via FFT and split into amplitude spectrum \(A\) and phase spectrum \(\Phi\). These are then superimposed with learnable positional encodings and sent to the Sequential State Space Model (S3M) for spectrum-associated context modeling. The modeled spectral components are transformed back to the spatial domain via IFFT and weighted by learnable importance vectors \(\alpha_L\), \(\alpha_M\), and \(\alpha_H\) before being fused into the enhanced feature \(P^*\) via an FFN. On one hand, \(P^*\) is provided to the RPN and detection head for target localization and classification. On the other hand, during training, RoI Align extracts GT and proposal features, which are then passed to the CPD branch for class prototype distillation.

The "high \(\rightarrow\) low scan," "alternate cross scan," and "low \(\rightarrow\) high scan" inside S3M in the figure below represent the specific directions of the multi-spectrum scanning strategy—which is the core of the second key design.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["Backbone + FPN"]
    B --> C["DoG Spectrum Disentanglement"]
    C --> D["Low-frequency Component"]
    C --> E["Middle-frequency Component"]
    C --> F["High-frequency Component"]
    D --> G["S3M: High→Low Scan<br/>Biased towards low-frequency aggregation"]
    E --> H["S3M: Alternate Cross Scan<br/>Balanced high & low-frequency info"]
    F --> I["S3M: Low→High Scan<br/>Biased towards high-frequency aggregation"]
    G --> J["Adaptive Weighted Fusion<br/>αL, αM, αH weights"]
    H --> J
    I --> J
    J --> K["FFN Fusion → P*"]
    K --> L["RPN + Detection Head"]
    K --> M["RoI Feature Extraction"]
    M --> N["CPD: Class Prototype Contrastive Distillation"]
    N -.->|"InfoNCE Loss"| L

Key Designs¶

1. Adaptive Spectrum Disentanglement (ASD): Full-Spectrum Cooperative Disambiguation

The most common failure mode in small object detection is misclassifying background textures as objects or submerging target signals under background noise, where different background interferences are distributed across different frequency bands. The design of ASD is to explicitly decompose features into three complementary frequency bands via DoG, perform context aggregation within each band, and finally adaptively fuse them. DoG decomposition is based on frequency-domain Gaussian filtering: applying FFT to feature \(P\), and then multiplying by three sets of filters: the low-frequency retains components within the Gaussian low-pass \(G(\omega, k\sigma)\), the middle-frequency extracts the ring area corresponding to the difference between \(G(\omega, \sigma)\) and \(G(\omega, k\sigma)\), and the high-frequency takes the residual outside \(1 - G(\omega, \sigma)\). The components \(P_{\text{low}}\), \(P_{\text{mid}}\), and \(P_{\text{high}}\) are obtained via IFFT. Each decomposed component undergoes the full pipeline of FFT \(\rightarrow\) amplitude/phase separation \(\rightarrow\) positional encoding \(\rightarrow\) S3M \(\rightarrow\) IFFT for intra-spectrum modeling. The reason for not simply applying convolution after DoG is that the noise patterns in the components decomposed by DoG are highly diverse (low-frequency contains smooth, large-area artifacts, while high-frequency contains scattered, isolated noise points). A sequence model capable of capturing long-range dependencies (rather than local convolution) is required to model them effectively. Finally, the three enhanced features are weighted by learnable vectors \(\alpha_L\), \(\alpha_M\), and \(\alpha_H\) (where each \(C\)-dimensional vector corresponds to the importance of a channel) and fused via FFN. The design of adaptive weights is key: the scale distribution of objects varies across different input images, so the contribution ratio of each spectrum should be dynamically adjusted.

2. Multi-Spectrum Scanning Strategy: Spectrum-Customized Mamba Context Modeling

S3M does not use conventional sequential scanning internally, but rather scanning sequences specifically designed for each spectrum. The motivation behind this is that state space models (SSMs like Mamba) process sequences sequentially, so hidden states are naturally dominated by adjacent elements, with the influence of distant elements decaying. This sequence sensitivity introduces spectral bias in the frequency domain. Leveraging this characteristic, we can design biased scanning paths for different spectra, steering SSM aggregation towards the crucial information of each spectrum. Specifically, the high-frequency component uses a spiral "low \(\rightarrow\) high" scanning order (from low-frequency areas to high-frequency areas) expanding outward along the radius from the spectral center, allowing high-frequency components to dominate the hidden states. The low-frequency component uses the opposite "high \(\rightarrow\) low" scanning (converging from the outside to the center) to let low-frequency components dominate the final representation. The middle-frequency component uses an "alternate cross scan," alternating between the inner and outer spectral rings during the scanning path, shuffling high and low-frequency signals to eliminate the inherent bias of SSM and achieve balanced modeling across spectra. Experiments show that replacing this spectrum-customized scanning strategy with generic sweeps, clockwise, or neighbor scans reduces AP by about 2-3 points (from 29.0 to 26.2-27.0), proving that the choice of scanning direction has a decisive impact on spectral modeling.

3. Class Prototype Distillation (CPD): Class-Level Feature Compression

Even if ASD cleans up the spectral features of each object, semantic discrepancies can still exist between intra-class targets (e.g., different "vehicle" instances can be far apart in the feature space due to differences in pose, occlusion, and lighting). CPD constructs semantic prototypes for each category in a shared embedding space, forcing all proposal features belonging to the same category to converge toward their corresponding prototype. In practice: during each training epoch, RoI features are extracted from ground-truth bounding boxes, passed through a Global Convolution layer (to compress spatial dimensions), and projected via MLP to obtain embedding vectors \(Z = [z_1, \dots, z_N]\). For each class \(c\), a prototype \(d_c^{(0)}\) is initialized as the empirical mean of all GT embeddings for that class. Subsequently, the prototype is refined using an affine-guided weighting mechanism (inspired by hierarchical affinity modeling): the cosine similarity between each GT embedding and its corresponding prototype \(d_c\) is calculated and converted into attention weights via softmax (with temperature \(\kappa\) controlling concentration) to re-estimate the prototype \(d_c^*\). The prototype library is updated across mini-batches via momentum to maintain global stability: \(d_c \leftarrow \alpha \cdot d_c + (1-\alpha) \cdot d_c^*\). Finally, RoI features are extracted from the proposal boxes generated by the RPN and projected into the same embedding space. An InfoNCE contrastive loss is applied to maximize the similarity between each proposal embedding and its corresponding class prototype while minimizing similarity to other class prototypes. The loss coefficient is \(\gamma=0.1\) and the temperature is \(\tau=0.07\). CPD is not a replacement for feature enhancement, but a complementary supervision mechanism—ASD cleans noise at the feature level, while CPD compresses intra-class distance at the representation level.

Loss & Training¶

The total loss consists of three parts: IoU loss (bounding box regression), cross-entropy classification loss, and prototype distillation loss, with a balancing coefficient \(\gamma=0.1\). Training is performed on a single A100: AI-TOD uses an input size of \(800 \times 800\), for 36 epochs; SODA-D and SODA-A use \(1200 \times 1200\), for 12 epochs. DoG parameters are set to \(\sigma=1.0\) and \(k=1.414\). The paper provides two variants: SFDNet (ResNet-50 backbone) and SFDNet* (Spatial-Mamba backbone, which is stronger).

Key Experimental Results¶

Main Results¶

AI-TOD Test Set (8 categories, 28,036 aerial images, average object size of only 12.8 pixels):

Method	Source	AP	AP50	AP75	APvt	APt	APs
DINO-Deformable-DETR	ICLR2023	23.2	56.6	15.4	9.9	23.1	29.3
DINO-5scale w/SET	CVPR2025	26.6	57.1	20.8	13.2	27.1	31.5
CFINet	ICCV2023	24.7	53.9	18.6	11.7	26.4	28.1
DNTR	TGRS2024	26.2	56.7	20.2	12.8	26.4	31.0
HS-FPN	AAAI2025	25.1	55.7	19.1	12.1	25.3	29.9
SFDNet (ResNet-50)	Ours	29.0	57.9	23.5	14.4	29.0	33.4
*SFDNet (Spatial-Mamba)**	Ours	31.7	64.9	25.6	17.6	32.8	36.1

It also achieves SOTA on SODA-D (24,828 high-resolution remote sensing images) and SODA-A (2,513 aerial images, oriented detection): SFDNet* reaches 34.2 AP on SODA-D (4.6 AP higher than HS-FPN) and 39.2 AP on SODA-A (6.0 AP higher than the previous SOTA GauCho).

Ablation Study¶

Contribution of each module on AI-TOD:

Configuration	AP	APt	APvt	APs	APm
Baseline (w/o ASD, w/o CPD)	24.8	24.8	9.3	30.3	38.2
+ ASD	28.6	28.8	13.8	33.2	39.2
+ CPD	27.2	27.6	12.0	32.2	38.3
Full (ASD+CPD)	29.0	29.0	14.4	33.4	40.8

Comparison of scanning mechanisms:

Strategy	AP	APvt	APt	APs
Sweep	26.7	13.6	27.1	30.6
Clockwise	27.0	13.1	27.4	31.2
Neighbor	26.2	12.3	26.6	29.7
Multi-Spectrum Scanning (Ours)	29.0	14.4	29.0	33.4

Key Findings¶

ASD is the largest contributor to performance: Adding ASD alone brings a +3.8 AP improvement (24.8 \(\rightarrow\) 28.6), while CPD provides an additional +0.4 AP boost on top of ASD (28.6 \(\rightarrow\) 29.0). Their improvement on extremely tiny objects (\(AP_{\text{vt}}\)) is particularly significant (baseline 9.3 \(\rightarrow\) Full 14.4, +5.1 points), indicating that spectral disambiguation is most sensitive for small targets.
Comparison of spectral aggregation mechanisms: S3M (SSM) significantly outperforms Attention (26.6 AP), VMamba (26.4 AP), and Convolution (26.3 AP), proving the unique advantage of sequential-sensitive SSM in the frequency domain.
Removing middle and high-frequency components hurts performance the most: Removing middle and high frequencies degrades performance more than removing low frequency, showing that the core information of small targets is concentrated in the higher frequency ranges. However, the low-frequency complement is still necessary (dropping low-frequency reduces \(AP_{\text{vt}}\) from 14.4 to 11.4).
Robustness of DoG parameters: Within the range of \(\sigma\) in 0.5-2.0 and \(k\) in 1.2-1.8, the AP fluctuation does not exceed 0.3, eliminating the need for meticulous parameter tuning.

Highlights & Insights¶

Deceptively Simple Background Disambiguation via Spectral Disentanglement: Previous background suppression methods either used fixed high-pass/low-pass filtering (losing information in the other band) or introduced adversarial training or attention mechanisms (computationally heavy and indirect). ASD transforms the problem into "decouple first, model separately, and adaptively merge", which preserves full spectral information while making the modeling goal of each sub-network more focused. This "decompose \(\rightarrow\) optimize separately \(\rightarrow\) fuse" paradigm is highly worthy of transfer to other tasks suffering from multi-source noise (e.g., image denoising, infrared object detection).
Scanning Direction as Inductive Bias: The sequence sensitivity of SSM is usually a drawback to overcome, but this paper turns it into a design space—by designing different scanning directions for different frequencies, the inherent order bias of SSM is converted into a controllable inductive bias. This concept can be generalized to any task that requires biased aggregation based on specific frequencies, patterns, or orientations.
CPD is a Lightweight Feature Alignment Scheme: Unlike relationship modeling based on graph networks or memory networks (which are computationally heavy with many hyperparameters), CPD only maintains a set of class prototype vectors and an InfoNCE loss, introducing virtually no inference overhead (CPD is removed during inference). This provides a lightweight, highly referenceable scheme for detection tasks requiring class-level constraints at the feature level.
Rational Design of the Two Variants: The ResNet-50 version is suitable for resource-constrained scenarios, while the Spatial-Mamba version pursues ultimate performance. Both achieve SOTA under their respective configurations.

Limitations & Future Work¶

The ASD module performs FFT/IFFT + 3-way S3M + weighted fusion on each FPN layer, which increases computational overhead compared to standard FPN, potentially limiting its deployment on resource-constrained edge devices. SFDNet* (with Spatial-Mamba backbone) has an even larger total computation cost, and the paper does not report specific FPS or parameter count comparisons.
CPD relies on ground-truth bounding boxes to construct prototypes. In scenarios with noisy or incomplete annotations (where some classes have very few instances in long-tailed distributions), prototype estimation may be unstable. The momentum update value \(\alpha\) may require class-by-class tuning under severe class imbalances.
The experiments only cover remote sensing/aerial photography scenarios (AI-TOD, SODA series) and have not yet been validated on general scenarios for small object detection (e.g., small pedestrians in CityPersons) or crowded scenarios (e.g., VisDrone). Whether full-spectrum disentanglement is equally effective for close-up small objects remains to be verified.
The DoG decomposition is still based on FFT operations with fixed parameters. Although robust to \(\sigma\) and \(k\), theoretically, the filters could be designed as learnable (to learn better spectral decomposition methods rather than manually setting the difference factor \(k\)).

vs HS-FPN: HS-FPN suppresses low frequencies based on a fixed high-pass filter driven by manual hyperparameters. SFDNet's ASD decomposes the entire spectrum into three components using DoG, models each using SSM, and adaptively fuses them, eliminating the need for manual frequency threshold design.
vs SET: SET performs spectral enhancement on specialized bands (focusing on signal enhancement), while SFDNet performs noise suppression across the entire spectrum (focusing on signal separation). The two paths are complementary and could theoretically be combined.
vs DNTR: DNTR uses contrastive learning along the top-down path of the FPN to suppress noise, but this noise suppression is performed in the spatial domain without frequency division; SFDNet refines the suppression granularity to the frequency domain.
vs Spatial-Mamba: Spatial-Mamba focuses on structure preservation in spatial neighborhoods. SFDNet uses it as a backbone (in SFDNet*), but the multi-spectrum scanning in S3M is conducted in the frequency domain. Their operations reside at different levels and can be used in combination.

Rating¶

Novelty: ⭐⭐⭐⭐ [The two core designs (spectrum-customized scanning + class prototype distillation) are both introduced to the SOD field for the first time. Each has some prior foundation, but their systematic combination is highly insightful]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Three standard benchmarks, five sets of ablations including module contribution, scanning mechanism, spectral component, aggregation function, and hyperparameters, with comprehensive quantitative and qualitative analyses]
Writing Quality: ⭐⭐⭐⭐ [The methodology is logically clear, and the motivation is well-justified, though there are many formulas, and the readability of the flowchart in Figure 2 has room for improvement]
Value: ⭐⭐⭐⭐ [The full-spectrum disambiguation idea can be generalized to other vision tasks severely affected by background interference. The code is open-source]