Slim-DETR: Real-Time Tiny Object Detection with Efficient Interaction and Gaussian Query

Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: Object Detection
Keywords: Tiny Object Detection, Real-Time Detection, Detection Transformer, Gaussian Query Selection, Efficient Interaction Encoder

TL;DR

To tackle encoder sequence length bottlenecks and severe query-object mismatch in DETR frameworks for tiny targets, Slim-DETR introduces an Efficient Interaction (EI) encoder with gather-and-inject foreground deformable attention alongside a size-decoupled Gaussian Target Guided (GTG) query selection strategy, achieving 33.9% AP on VisDrone with real-time inference on both RTX 3090 (24.4 ms) and Jetson AGX Orin (40 FPS).

Background & Motivation

Tiny object detection is indispensable in resource-constrained edge scenarios such as drone-based scene analysis, remote sensing reconnaissance, traffic monitoring, and disaster management. Unlike generic objects, tiny targets span extremely few pixels (typically under \(32 \times 32\) or even \(16 \times 16\) pixels) and lack rich texture and clear contours. Consequently, detectors must heavily exploit high-resolution, low-level features (such as the backbone's B2 stage) containing rich local geometric details. However, when conventional end-to-end Transformer detectors (DETRs) introduce high-resolution multi-scale feature interactions, the quadratic scaling of attention over drastically expanded sequence lengths creates severe computational and latency bottlenecks in the encoder, precluding real-time edge deployment. While real-time variants like RT-DETR discard low-level attention interactions to maintain speed, this omission discards fine-grained spatial cues, causing extensive missed detections for tiny targets.

Beyond the encoder efficiency bottleneck, standard DETR architectures suffer from a critical query-object mismatch during the query selection phase following feature encoding. Mainstream criteria rely on heuristic scoring combining bounding box IoU and classification confidence, which inherently favors large targets and salient background textures. Because tiny targets exhibit virtually zero prior bounding box overlap and weak initial classification scores, candidate queries are overwhelmingly allocated to large objects and background regions. This allocation imbalance deprives tiny objects of qualified queries, discarding minute targets before decoder iterative refinement begins.

To overcome these two structural bottlenecks, this paper proposes Slim-DETR, a streamlined end-to-end framework tailored for real-time tiny object detection. Core idea: build an Efficient Interaction (EI) encoder using a gather-and-inject paradigm that filters background noise via CBAM and gating units, restricts sparse deformable attention exclusively to foreground tokens, and connects directly to cross-scale fusion; paired with a size-decoupled Gaussian Target Guided (GTG) query selection strategy that models target likelihoods as continuous Gaussians to concentrate high-quality queries on tiny targets.

Method

Overall Architecture

Slim-DETR adopts an end-to-end pipeline designed for high fidelity and minimal edge latency. Given an input image, a convolutional backbone (e.g., ResNet-18 or ResNet-50) extracts multi-scale feature maps \(\{B_2, B_3, B_4\}\). Instead of employing multi-layer dense attention across all scales, features enter the Efficient Interaction (EI) encoder governed by a gather-and-inject paradigm. Multi-level features are first spatially aligned, enhanced by CBAM, modulated by a gating unit to suppress background clutter, and fused into a unified global representation. The Foreground Sampling Deformable Attention (FSDA) module then isolates potential foreground tokens to execute sparse deformable attention, and the updated tokens are embedded back into their spatial positions before being directly injected into lightweight cross-scale fusion blocks. Finally, the Gaussian Target Guided (GTG) module predicts size-decoupled Gaussian distribution maps to allocate concentrated, high-quality initial queries to tiny objects, feeding a compact Transformer decoder to output final bounding boxes and category logits.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Scale Backbone Inputs<br/>{B2, B3, B4}"] --> B["Multi-Level Gathering & Gate Modulation<br/>Resolution alignment + CBAM spatial/channel attention + Gate suppression"]
    B --> C["Foreground Sampling Deformable Attention (FSDA)<br/>Top-k foreground mask sampling + Sparse interaction + In-place embedding"]
    C --> D["Direct Cross-Scale Semantic Injection<br/>Inject enhanced global features into level-wise Cross Fusion"]
    D --> E["Gaussian Target Guided Query Selection (GTG)<br/>3-layer CNN Gaussian heatmap prediction + Size-decoupled allocation"]
    E --> F["Compact Transformer Decoder<br/>4-layer efficient decoding for final class & box prediction"]

Key Designs

1. Gather-and-Inject Paradigm with Gate Modulation: bypassing indirect propagation and suppressing background noise Standard feature pyramid networks (FPN) suffer from lengthy top-down and bottom-up pathways where subtle semantic signals from tiny objects are easily washed out across successive convolution layers. Slim-DETR avoids multi-layer dense attention cascades by projecting multi-scale features \(\{B_2, B_3, B_4\}\) to a unified intermediate resolution matching \(B_3\). To compensate for the restricted receptive fields and semantic deficit of low-level features, CBAM modules are applied across branches to capture complementary channel ("what") and spatial ("where") cues. Subsequently, a gate unit converts the CBAM-refined features into a spatial gating signal \(G\). This gate assigns near-zero activations to texture-heavy but semantically ambiguous background regions (e.g., ripples, foliage, or road glare), suppressing noise and spotlighting tiny object regions before a lightweight CNN-based Fusion (CF) module incorporating depthwise-separable RepPD blocks produces a compact global feature map \(F_G\).

2. Foreground Sampling Deformable Attention (FSDA): sparse semantic interaction without cumulative layer miss rates Processing full high-resolution sequences through attention blocks incurs prohibitive latency. Existing sparse DETRs (e.g., Salience-DETR or Sparse-DETR) filter tokens hierarchically layer by layer, which leads to cumulative error propagation where a missed tiny target at an early layer is permanently lost. FSDA circumvents this by performing single-stage foreground extraction on the unified global feature map \(F_G\). Guided by an auxiliary binary cross-entropy loss, an auxiliary head predicts a foreground probability mask \(\mathbf{M}_s \in [0, 1]^{H \times W}\). The top-\(k\) foreground query indices \(\mathcal{S}_k\) are dynamically sampled according to ratio \(\rho\):

\[\mathcal{S}_k = \left\{ m \mid \mathbf{M}_s(m) \in \text{top-}k(\text{sort}(\mathbf{M}_s)) \right\}, \quad \text{where } k = \rho HW\]

Only the sampled foreground tokens \(f_s = F_G[\mathcal{S}_k]\) act as queries to interact with the full global feature map through deformable attention. To preserve 2D spatial topology and prevent spatial discontinuity from disrupting subsequent convolutions, the updated foreground features are embedded directly back into their original spatial coordinates:

\[\mathbf{F}'_{Gm} = \begin{cases} \text{DAttention}\left( f_s + \text{pos}_m, F_G + \text{pos}, F_G \right), & m \in \mathcal{S}_k \\ F_{Gm}, & m \notin \mathcal{S}_k \end{cases}\]

This enhanced feature map \(F'_G\) is then directly injected into all levels of the lightweight cross-fusion module via skip connections, providing immediate global semantic context without indirect layer-by-layer traversal.

3. Size-Decoupled Gaussian Target Guided Query Selection (GTG): scale-invariant dynamic query concentration Conventional query selection relies on a composite score of classification confidence and IoU overlap. Because tiny objects have minuscule spatial footprints and negligible prior bounding box intersection, their selection scores are overwhelmed by large objects and background textures. GTG replaces heuristic ranking with continuous 2D Gaussian probability distributions. For ground-truth bounding box \(\mathcal{D}_{\text{Bbox}} = (x_l, y_l, w_l, h_l)\) at scale level \(l\), the target Gaussian response is modeled as:

\[Y_l^{(i, j)} = \exp\left( -\frac{(p_i^l - x_l)^2 + (p_j^l - y_l)^2}{2\sigma_l^2} \right), \quad (i, j) \in \mathcal{D}_{\text{Bbox}}\]

The Gaussian peak is normalized to 1 regardless of object scale, guaranteeing scale parity. Responses smoothly decay toward zero at object boundaries and remain strictly zero across background pixels. Crucially, GTG decouples prediction across scales: targets \(\le 32 \times 32\) are predicted via \(G_2\), targets between \(32 \times 32\) and \(64 \times 64\) via \(G_3\), and larger targets via \(G_4\). Queries are allocated in proportion to the integrated Gaussian response of each branch, concentrating high-confidence initial queries directly onto tiny targets.

Loss & Training

Slim-DETR is optimized end-to-end using a multi-task loss objective:

\[\mathcal{L}_{\text{total}} = \lambda_m \mathcal{L}_m + \lambda_G \mathcal{L}_G + \lambda_{\text{box}} \mathcal{L}_{\text{box}} + \lambda_{\text{cls}} \mathcal{L}_{\text{cls}}\]

Here, \(\mathcal{L}_m\) denotes the auxiliary foreground mask supervision loss trained via Binary Cross-Entropy (BCE) against binary ground-truth bounding box masks. \(\mathcal{L}_G\) supervises the 3-layer CNN Gaussian prediction branch using a combination of Gaussian Focal Loss (\(\alpha = 2, \beta = 4\)) and balanced MSE regression loss. Bounding box regression \(\mathcal{L}_{\text{box}}\) (L1 and GIoU loss) and classification \(\mathcal{L}_{\text{cls}}\) follow the standard RT-DETR formulation. Loss weighting coefficients are set to \(\lambda_m = 0.01\), \(\lambda_G = 0.5\), \(\lambda_{\text{box}} = 1\), and \(\lambda_{\text{cls}} = 1\). The model is trained on 8 NVIDIA GeForce RTX 3090 GPUs using AdamW with an initial learning rate of 0.0006.

Key Experimental Results

Main Results

On the drone benchmark VisDrone (\(640 \times 640\)) and the extreme tiny-object aerial benchmark AI-TOD-V2 (\(800 \times 800\)), Slim-DETR establishes a superior trade-off between detection accuracy and inference latency:

Dataset	Method	AP	\(\text{AP}_{50}\)	\(\text{AP}_{75}\)	Params (M)	GFLOPs	Latency (ms)
VisDrone val	QueryDet [CVPR'22]	19.6	35.7	19.0	-	-	285.7
VisDrone val	YOLOv8-L	26.6	43.5	27.4	43.6	165	23.8
VisDrone val	YOLOv11-L	26.2	42.9	26.9	25.3	87	15.6
VisDrone val	YOLOv12-L	26.4	43.3	27.0	26.3	89	16.4
VisDrone val	RT-DETR-R18	26.7	44.6	25.9	20.0	60	13.8
VisDrone val	Salience-DETR	29.4	46.2	29.1	-	201	82.2
VisDrone val	RT-DETR-R50	30.3	50.8	30.3	42.0	136	19.6
VisDrone val	DEIM-R50 [CVPR'25]	30.4	50.9	30.5	42.0	136	19.1
VisDrone val	D-FINE-R50	32.0	52.1	33.2	40.7	117	18.0
VisDrone val	Slim-DETR-R18 (Ours)	32.4	52.5	33.5	5.9	51	14.3
VisDrone val	Slim-DETR-R50 (Ours)	33.9	55.2	36.9	15.7	131	24.4

Performance on the AI-TOD-V2 test set highlights substantial gains on very tiny targets (\(\text{AP}_{vt}\) and \(\text{AP}_t\)):

Dataset	Method	AP	\(\text{AP}_{50}\)	\(\text{AP}_{75}\)	\(\text{AP}_{vt}\)	\(\text{AP}_t\)	\(\text{AP}_s\)	Latency (ms)
AI-TOD-V2	Deformable-DETR	18.9	50.0	10.5	6.5	17.6	25.3	66.7
AI-TOD-V2	DINO-DETR	23.2	56.6	15.4	9.9	23.1	29.3	92.1
AI-TOD-V2	Salience-DETR	23.4	56.4	15.7	9.3	23.0	29.3	158.9
AI-TOD-V2	RT-DETR-R50	27.6	59.7	17.5	12.7	25.3	32.0	38.4
AI-TOD-V2	DQ-DETR [ECCV'24]	30.2	68.6	22.3	15.3	30.5	36.5	-
AI-TOD-V2	Slim-DETR-R50 (Ours)	33.6	69.7	27.9	16.4	34.0	38.5	39.4

Ablation Study

1. Encoder Architecture and Latency Ablation (VisDrone, evaluated under high-resolution \(\{B_2, B_3, B_4\}\) inputs):

Configuration / Variant	Modules	Latency (ms)	AP	\(\text{AP}_{50}\)	\(\text{AP}_{75}\)	\(\text{AP}_s\)	Note
Baseline	Single-Scale Encoder (SSE)	100.00	30.7	50.4	30.5	21.7	High resolution causes severe attention latency
Variant A	Multi-Scale Encoder (MSE)	166.66	31.3	51.9	31.7	22.8	Sequence explosion degrades speed to 166.66 ms
Variant B	Gather-and-Inject	28.57	33.1	54.0	33.7	24.6	3.5× acceleration and +2.4% AP gain over Baseline
Variant C (EI)	Gather-and-Inject + FSDA	24.39	33.3	54.3	34.2	24.9	Foreground-only attention yields 4.1× speedup and highest AP

2. Decoder Depth and GTG Query Selection Ablation (Slim-DETR-R50, VisDrone val set):

Query Selection	Decoder Layers (\(N\))	AP	\(\text{AP}_{50}\)	\(\text{AP}_s\)	Params (M)	GFLOPs	Latency (ms)	Note
Uncertainty-minimal [RT-DETR]	6	33.3	54.3	24.7	16.3	128	24.39	Conventional IoU & confidence ranking
GTG (Gaussian)	6	33.5	54.5	24.9	17.2	137	27.03	Better query quality yields +0.2% AP
GTG (Gaussian)	5	33.7	54.7	25.0	16.4	134	26.31	Removing redundant layers improves performance
GTG (Gaussian)	4	33.9	55.2	25.3	15.7	131	24.41	Optimal trade-off: fewer layers offset CNN head cost, peak AP
GTG (Gaussian)	3	33.2	53.9	24.2	14.9	128	23.25	Under-parameterized decoding degrades localization

3. Edge Platform Deployment (NVIDIA Jetson AGX Orin, TensorRT FP16, VisDrone input \(416 \times 416\)): - YOLOv8-M: 68.68 ms (15 FPS), 24.6% AP - YOLOv12-M: 90.12 ms (11 FPS), 24.8% AP - RT-DETR-416: 35.98 ms (27 FPS), 23.5% AP - Slim-DETR-416: 25.12 ms (40 FPS), 25.4% AP (outperforms RT-DETR by +1.9% AP while accelerating throughput from 27 FPS to 40 FPS).

Key Findings

• High-resolution inputs are essential for tiny targets, but require gather-and-inject decoupling: Shifting input features to \(\{B_2, B_3, B_4\}\) in a conventional multi-scale encoder spikes latency to 166.66 ms. The gather-and-inject architecture cuts redundant cross-layer attention paths, reducing latency to 28.57 ms, and FSDA further reduces it to 24.39 ms (a 4.1× overall speedup) while boosting \(\text{AP}_s\) by 3.2%.
• High-quality initial queries mitigate reliance on deep decoders: Because GTG anchors queries tightly around tiny target centroids, the decoder does not require 6 iterative refinement layers. Pruning the decoder to 4 layers lowers overall parameter count (from 16.3M to 15.7M) and achieves peak accuracy (33.9% AP), confirming that high-quality initial queries allow shallower decoders to thrive.
• Sampling ratio matches dataset foreground proportions: Detection accuracy reaches an empirical plateau when \(\rho \ge 0.25\) on VisDrone and \(\rho \ge 0.1\) on AI-TOD-V2. Increasing \(\rho\) beyond these values adds redundant background computation without accuracy gains.

Highlights & Insights

• Continuous Gaussian distributions replace brittle IoU thresholds: By normalizing Gaussian peak probabilities to 1 and decaying responses smoothly to target boundaries, GTG eliminates the intrinsic scale bias of IoU-based selection and ensures scale-invariant query allocation.
• In-place foreground embedding maintains spatial topology: Unlike sparse Transformers that permanently discard background tokens and disrupt 2D coordinate structure, FSDA applies attention exclusively to foreground tokens and re-embeds them into the global canvas, preserving spatial integrity for downstream convolutions.
• Ultra-compact model outperforming bulky SOTA: With only 5.9M parameters, Slim-DETR-R18 reaches 32.4% AP on VisDrone, comfortably outperforming 40M+ models like RT-DETR-R50 (30.3% AP) and DINO-DETR (29.4% AP) at a fraction of the parameter footprint.

Limitations & Future Work

• Static sampling ratio across varying scene densities: Sampling ratio \(\rho\) is configured statically per dataset (\(\rho=0.25\) for VisDrone, \(\rho=0.1\) for AI-TOD-V2). In video streams where object density shifts dynamically from empty horizons to congested crowds, fixed ratios risk either foreground clipping or redundant background processing.
• Rigid scale-decoupling cutoffs: GTG uses rigid thresholds of 32 and 64 pixels to assign targets to distinct prediction branches (\(G_2, G_3, G_4\)), which could introduce boundary artifacts for objects near cutoff thresholds. Continuous scale-weighting mechanisms warrant future investigation.

Related Work & Insights

• vs RT-DETR [CVPR'24]: RT-DETR drops high-resolution feature attention to preserve speed, causing unrecoverable loss of tiny targets, while relying on indirect FPN cross-scale transfers. Slim-DETR retains fine details via gather-and-inject foreground attention, improving VisDrone AP by +5.7% (R18) and +3.6% (R50).
• vs DQ-DETR [ECCV'24]: While DQ-DETR introduces density maps to adjust query numbers dynamically, its query selection relies on traditional confidence scores dominated by large objects. Slim-DETR's GTG normalizes probability peaks to 1, providing true scale-invariant query initialization.

Rating

• Novelty: ⭐⭐⭐⭐ [Well-motivated gather-and-inject foreground attention coupled with scale-decoupled Gaussian query selection]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across VisDrone and AI-TOD-V2 benchmarks, thorough encoder/decoder ablations, and real-time validation on Jetson Orin]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and coherent narrative structure]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, high-throughput end-to-end detection baseline for resource-constrained edge and drone platforms]