RiO-DETR: DETR for Real-time Oriented Object Detection

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/RicePasteM/RiO-DETR
Area: Object Detection
Keywords: Oriented Object Detection, Real-time Object Detection, Detection Transformer, Aerial Object Detection, Orthogonal Attention

TL;DR

Addressing high latency, semantic-geometric entanglement, and periodic optimization instability in prior oriented DETRs, RiO-DETR introduces geometry-decoupled query encoding, rotation-rectified orthogonal attention, decoupled periodic refinement, and oriented dense O2O supervision, establishing the first end-to-end real-time oriented detection transformer with 2.7–29.9 ms latency and 78.4%–81.8% AP50 on DOTA-1.0.

Background & Motivation

Oriented object detection (OBB) extends conventional horizontal bounding boxes to arbitrary orientations, serving as a foundational technology for satellite Earth observation, aerial drone inspection, and oriented scene text parsing. Driven by the recent surge in edge computing and low-altitude economy applications, academic and industrial priorities have shifted from pursuing peak accuracy to mastering the speed–accuracy trade-off. Within the convolutional neural network (CNN) paradigm, oriented versions of YOLO (such as YOLO26-obb) and RTMDet-R have set robust baselines for high-throughput edge deployment. Concurrently, real-time detection transformers such as RT-DETR and its variants have achieved remarkable latency–accuracy frontiers in horizontal detection via lightweight hybrid encoders and uncertainty-minimal query selection, eliminating hand-crafted NMS post-processing. Nevertheless, oriented detection transformers have remained largely confined to heavy academic architectures; prominent oriented DETRs such as ARS-DETR, Oriented-DETR, and RHINO-DETR suffer from inference latencies ranging from 150 to 300 ms, excluding them from practical real-time deployment.

A deeper diagnosis reveals that this efficiency bottleneck is not caused by inadequate parameter capacity, but rather stems from fundamental architectural mismatches between OBB geometry and standard DETR operations. The primary issue lies in semantic–geometric coupling and feature collapse: traditional oriented DETRs routinely embed the full 5D tuple \((cx, cy, w, h, \theta)\) into positional query embeddings. However, unlike spatial center coordinates, the orientation \(\theta\) exhibits rotational periodicity and admits equivalent parameterizations (e.g., swapping width and height while shifting \(\theta\) by \(\pi/2\)). The ground-truth angle is fundamentally anchored to semantic appearance cues such as texture flow and canonical object heading rather than pure Euclidean coordinates; feeding an uncalibrated noisy angle prior into positional embeddings distorts the spatial sampling grid during early training. Furthermore, aligning all attention heads along the primary long axis causes longitudinal feature collapse, ignoring lateral contextual structures. The second issue is the periodicity mismatch in box refinement: standard DETRs perform Euclidean additive updates within the inverse-sigmoid domain, which breaks down on the circular manifold of angles and triggers severe gradient spikes and boundary seam artifacts across the \(0 \leftrightarrow \pi\) transition. Lastly, the expanded degrees of freedom significantly enlarge the bipartite matching search space, resulting in slow training convergence that existing horizontal dense supervision methods fail to resolve due to lack of angular diversity.

To resolve these architectural bottlenecks, the authors dispense with ad-hoc external angle heads and heavy sampling operators, presenting RiO-DETR as a native real-time end-to-end oriented detection transformer. Core idea: decouple angle prediction from positional queries into a semantic content-driven mechanism, paired with rotation-rectified orthogonal attention for bi-directional feature sampling, bounded coarse-to-fine periodic refinement, and quadrant-rotated dense supervision, fully unlocking the speed and accuracy potential of DETR for real-time oriented object detection.

Method

Overall Architecture

RiO-DETR builds upon an efficient hybrid encoder combined with a single/multi-scale refinement decoder. Given an input high-resolution aerial image, an HGNetv2 backbone extracts multi-scale visual features, which are subsequently aggregated through a cross-scale hybrid encoder. In the decoder stage, object queries are partitioned into positional and content queries: positional queries only encode 4D spatial coordinates \((cx, cy, w, h)\), preserving rotation invariance, while orientation is inferred from content feature semantics. Inside multi-head deformable attention, attention heads are split into mutually orthogonal axial and lateral sampling groups. The decoder progressively refines box coordinates and orientations using a decoupled periodic refinement mechanism, optimized during training via Oriented Dense O2O supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Aerial Image"] --> B["HGNetv2 Backbone & Hybrid Encoder"]
    B --> C["Geometry-Decoupled Query Encoding<br/>Positional queries encode only (cx, cy, w, h)"]
    C --> D["Rotation-Rectified Orthogonal Attention<br/>Axial heads (θ) and orthogonal heads (θ + π/2)"]
    D --> E["Decoupled Periodic Refinement<br/>Bounded tanh layer-wise decay & modulo normalization"]
    E --> F["Oriented Dense O2O Training<br/>Independent quadrant rotations for angular diversity"]
    F --> G["NMS-free End-to-End Output (cx, cy, w, h, θ, class)"]

Key Designs

1. Geometry-Decoupled Query Encoding: preventing noisy angle priors from misleading spatial attention

Prior oriented DETRs inject the full 5-dimensional tuple into positional queries. However, because orientation is periodic and exhibits equivalent parameterizations (e.g., \((w, h, \theta)\) vs. \((h, w, (\theta + \pi/2) \bmod \pi)\)), \(\theta\) represents a canonical convention dictated by semantic appearance rather than a strict geometric coordinate. In early training epochs, injecting a fluctuating reference orientation \(\theta_{ref}\) into positional embeddings enforces an inaccurate geometric constraint that distorts attention sampling grids. RiO-DETR decouples orientation from positional queries by restricting the coordinate encoder \(\phi(\cdot)\) strictly to the 4-dimensional spatial domain:

\[Q_{pos} = \phi(p_{ref}[..., :4]) = \text{MLP}(\text{PE}(p_{ref}[..., :4]))\]

This maintains strict rotation invariance for \(Q_{pos}\), dedicating it entirely to coarse spatial localization \((cx, cy, w, h)\). Concurrently, orientation cues are latently modeled within learnable content embeddings \(Q_{content}\), compelling the model to extract rotation hints from image textures, component layouts, and aspect-ratio indicators.

2. Rotation-Rectified Orthogonal Attention: orthogonal sampling to eliminate longitudinal feature collapse

While query decoupling stabilizes positional optimization, feature extraction still requires deformable sampling points aligned with object boundaries. Prior oriented transformers align all attention heads with the predicted primary orientation, causing feature collapse where the network disproportionately attends to longitudinal details while ignoring lateral context such as wings, hulls, and lateral boundaries. RiO-DETR introduces rotation-rectified orthogonal attention with zero additional parameter overhead. For an even number of attention heads \(H\), the heads are evenly bifurcated into two groups:

\[\theta(h) = \begin{cases} \theta, & \text{if } h \le \frac{H}{2} \\ \theta + \frac{\pi}{2}, & \text{if } h > \frac{H}{2} \end{cases}\]

For query \(q\) and sampling point \(k\), the learned offset \(\Delta p_{qk}\) scaled by \(s_q\) is rotated by the head-specific transformation matrix \(R(\theta(h))\):

\[S(p_q, \Delta p_{qk}) = p_q + R(\theta(h))(\Delta p_{qk} \odot s_q)\]

\[R(\theta(h)) = \begin{bmatrix} \cos(\theta(h)) & -\sin(\theta(h)) \\ \sin(\theta(h)) & \cos(\theta(h)) \end{bmatrix}\]

The first half of the heads captures longitudinal continuity along the object's major axis, while the second half samples orthogonally to capture width boundaries and transverse structures, significantly improving localization for extreme aspect-ratio targets like bridges and large ships.

3. Decoupled Periodic Refinement: bounded coarse-to-fine updates with shortest-path periodic loss

Standard DETR decoders update bounding boxes via unconstrained additive offsets in inverse-sigmoid space. While valid in Euclidean space, orientation \(\theta\) resides on the cyclic manifold \(S^1\). Direct Euclidean updates cause numerical discontinuities near angular boundaries (\(0 \leftrightarrow \pi\)), where geometrically identical or near-identical angles produce massive gradients that destabilize training. Decoupled periodic refinement aligns both update step and loss formulation with periodic geometry. At layer \(i\), the predicted angle offset \(\Delta \theta_i\) is bounded by \(\tanh(\cdot)\) and scaled by a layer-wise decaying factor \(\alpha_i = \alpha_0^{-i}\) (\(\alpha_0 > 1\)):

\[\theta_{raw} = \theta_{ref} + \tanh(\Delta \theta_i) \cdot \alpha_i\]

\[\theta_{new} = \begin{cases} (\theta_{raw} \bmod \pi) + \pi, & \text{if } (\theta_{raw} \bmod \pi) < 0 \\ (\theta_{raw} \bmod \pi), & \text{otherwise} \end{cases}\]

Early layers execute larger corrective rotations, whereas deeper layers perform fine-grained adjustments without overshooting. To prevent Euclidean over-penalization across seams, the angular loss is formulated as a Shortest-Path Periodic L1 Loss:

\[\mathcal{L}_{angle} = \min(|\theta_{pred} - \theta_{tgt}|, \pi - |\theta_{pred} - \theta_{tgt}|)\]

ensuring that optimization gradients strictly flow along the shortest arc on the circle.

4. Oriented Dense O2O: quadrant-independent rotations for accelerated angle convergence

The enlarged search space in rotated bipartite matching causes slow convergence during early training. While Dense O2O stitches four identical sub-images to increase ground-truth density, the identical orientations provide zero rotational diversity. Oriented Dense O2O applies an independent discrete rotation \(\theta_{rot} \in \{0^\circ, 90^\circ, 180^\circ, 270^\circ\}\) to each quadrant before composite mosaic assembly. This zero-cost data organization presents varied orientations of identical categories within a single forward pass, stabilizing Hungarian matching and accelerating angle branch convergence by approximately 30%.

Loss & Training

The bipartite matching cost combines classification and rotation-aware geometry:

\[C_{match} = \lambda_{cls} C_{focal} + \lambda_{kld} C_{kld} + \lambda_{hausdorff} C_{hausdorff}\]

incorporating Gaussian distribution-based KLD distance and Hausdorff metric to align predictions with ground-truth orientation and shape. The overall training loss integrates focal loss, KLD loss, and the shortest-path periodic angle loss:

\[\mathcal{L} = \lambda_{cls} \mathcal{L}_{focal} + \lambda_{kld} \mathcal{L}_{kld} + \lambda_{angle} \mathcal{L}_{angle}\]

Combined with D-FINE fine-grained matching and Dense O2O auxiliary supervision, the entire model achieves high-precision NMS-free end-to-end inference.

Key Experimental Results

Main Results

Under the single-scale training and evaluation protocol on DOTA-1.0, RiO-DETR variants span from Nano to XLarge, comprehensively outperforming prior real-time CNN detectors and non-real-time DETR models on an NVIDIA T4 GPU (TensorRT FP16).

Method	Backbone	#Params	FLOPs	Latency (ms)	AP50 (%)
LSKNet-S	LSKNet-S	31.0M	161G	203.5	77.5
PKINet-S	PKINet-S	30.8M	190G	359.7	78.4
Strip R-CNN-S	StripNet-S	30.5M	159G	241.9	80.1
ARS-DETR	Swin-T	41.9M	431G	303.6	75.5
RHINO-DETR	Swin-T	50.8M	609G	242.6	79.4
Oriented-DETR	Swin-T	57.7M	309G	235.5	79.8
YOLO26n-obb	YOLO26n	2.5M	14G	2.8	77.7
YOLO26s-obb	YOLO26s	9.8M	55G	4.9	79.7
YOLO26m-obb	YOLO26m	21.2M	183G	10.2	80.0
YOLO26x-obb	YOLO26x	57.6M	517G	30.5	80.4
RiO-DETR-n (Ours)	HGNet-B0	4.0M	17G	2.7	78.4
RiO-DETR-s (Ours)	HGNet-B2	8.2M	53G	5.2	80.3
RiO-DETR-m (Ours)	HGNet-B2	18.6M	158G	8.8	80.9
RiO-DETR-l (Ours)	HGNet-B4	27.5M	230G	13.4	81.7
RiO-DETR-x (Ours)	HGNet-B5	62.5M	527G	29.9	81.8

Ablation Study

On DIOR-R, a step-by-step ablation using RiO-DETR-m demonstrates the quantitative contributions of individual architectural innovations:

Stage / Configuration	#Params	FLOPs	Latency (ms)	AP50 (%)	Gain
Oriented RT-DETRv2 (Scratch Baseline)	18.61M	97.06G	5.04	70.35	Base
+ Hausdorff Matching & KLD Loss	18.61M	97.06G	5.04	72.86	+2.51
+ Universal Matching Strategy (D-FINE)	18.61M	97.06G	5.04	73.33	+0.47
+ Dense O2O (DEIM Strong Baseline)	18.61M	97.06G	5.04	73.47	+0.14
+ Geometry-Decoupled Query Encoding	18.59M	97.01G	5.04	74.18	+0.71
+ Rotation-Rectified Orthogonal Attention	18.67M	97.18G	5.10	74.74	+0.56
+ Decoupled Periodic Refinement	18.67M	97.18G	5.10	75.46	+0.72
+ Oriented Dense O2O (Full RiO-DETR-m)	18.67M	97.18G	5.10	75.73	+0.27

Component-specific analysis reveals: 1. Query Geometry Formulation: Encoding center \((cx, cy)\) alone yields 72.56% AP50; adding angle \(\theta\) drops accuracy to 72.34%. Adding size parameters \((w, h)\) yields 73.57%, while excluding angle to rely purely on spatial coordinates \((cx, cy, w, h)\) achieves the optimal 73.81% prior to orthogonal attention. 2. Orthogonal Head Allocation: Directing all heads along \(\theta\) yields 73.81%; an asymmetric 6:2 split improves to 74.02%; symmetric 4:4 orthogonal splitting achieves the peak of 74.18%, whereas a 4-direction split drops to 73.95% due to capacity dilution. 3. Periodic Refinement Synergy: SP-L1 alone improves AP50 to 74.32%, while periodic update alone drops slightly to 74.05% due to gradient misalignment; their combination yields 74.74%, confirming the necessity of topological and metric alignment.

Key Findings

• Decoupling Delivers Pure Accuracy Gains: Eliminating \(\theta\) from positional queries prevents early geometric misguidance, speeding up convergence without degrading orientation sensitivity.
• Orthogonal Attention Resolves Extreme Aspect Ratios: For objects with large aspect ratios like harbors and bridges, lateral attention sampling provides essential transverse boundary cues, leading to marked recall gains.
• Discrete Rotations Outperform Continuous Angles in Dense O2O: Discrete four-way rotations achieve 73.88% AP50 at epoch 60 compared to 73.66% at epoch 62 for continuous rotations, as discrete rotations prevent equivalent parameterization jitter on near-square targets.

Highlights & Insights

• Demoting Geometric Angle Priors to Semantic Features: Unlike prior works that inflate query embeddings with multi-dimensional geometry, this paper recognizes orientation as an appearance-governed canonical property, decoupling it to resolve early optimization conflicts.
• Zero-Cost Orthogonal Attention: Eliminating feature collapse along object major axes by simply rotating existing attention heads by \(\pi/2\), avoiding additional deformable sampling points or high-latency operators.
• First True Real-Time NMS-Free Oriented DETR: Achieving 2.7 ms (Nano) and 29.9 ms (XLarge) latencies while eliminating heuristic Rotated NMS, significantly simplifying edge deployment on TensorRT.

Limitations & Future Work

• Absence of a Domain-Tailored Remote Sensing Backbone: RiO-DETR relies on general-purpose HGNetv2 backbones; specialized lightweight architectures tailored for dense clutter and tiny remote sensing objects could yield further gains.
• Residual Ambiguity on Square-Like Objects: Symmetrical objects (e.g., storage tanks) still present intrinsic 90-degree rotational ambiguity, occasionally causing minor orientation jitter.
• Future Directions: Investigating multi-modal vision-language aerial priors and adaptive geometric heads for fine-grained tiny rotated targets.

Related Work & Insights

• vs RHINO-DETR / ARS-DETR / Oriented-DETR: Prior academic oriented DETRs focus on complex attention schemes and high-dimensional queries, resulting in 50M+ parameters and >200 ms latency; RiO-DETR achieves comparable or superior accuracy while cutting inference latency by up to 10× (down to 2.7 ms).
• vs YOLO26-obb / RTMDet-R: Conventional oriented CNN detectors rely on dense anchor predictions and computationally heavy Rotated NMS post-processing; RiO-DETR demonstrates that pure transformer architectures can match or exceed YOLO speeds while providing clean end-to-end NMS-free deployment.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First end-to-end real-time oriented detection transformer with principled orthogonal attention and geometric decoupling.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across DOTA-1.0 (single/multi-scale), DIOR-R, and FAIR-1M-2.0 with meticulous ablation experiments.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid formulation addressing geometric topology mismatches and DETR architectural bottlenecks.
• Value: ⭐⭐⭐⭐⭐ Offers an invaluable real-time NMS-free baseline for edge robotics and aerial remote sensing perception.

Related Papers

• [ICLR 2026] RF-DETR: Neural Architecture Search for Real-Time Detection Transformers
• [ECCV 2026] RT-SDGOD: Real-Time Single-Domain Generalized Object Detection
• [ICLR 2026] DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
• [ECCV 2026] A Simple Baseline with Placement Prior for Point-Supervised Oriented Object Detection
• [CVPR 2025] MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism