Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Riiluo/FressDet
Area: Object Detection
Keywords: multispectral object detection, rotation-equivariant neural networks, implicit spectral warping, spectral-spatial learning, oriented object detection
TL;DR¶
To resolve the bottlenecks of discrete spectral processing, scale-dependent cross-level fusion degradation, and geometric disorientation in multispectral object detection, FressDet introduces a fully rotation-equivariant framework featuring order-preserving continuous spectral implicit warping and dual-prototype consistency weighting, establishing a new SOTA with 93% fewer parameters.
Background & Motivation¶
Multispectral object detection (MOD) enhances visual localization and recognition by combining spatial structural information with discriminative spectral signatures across varied wavelengths, presenting notable advantages under low illumination, shadow, camouflage, and dense aerial scenarios. However, conventional multispectral detectors suffer from two fundamental representation defects. First, most models treat spectral bands as disconnected discrete channels via dimension reduction, band selection, or standard attention mechanisms, failing to capture the intrinsic physical continuity and strict monotonic order along the wavelength dimension. Second, in multi-scale feature pyramid networks (such as FPN or PAFPN), cross-level feature fusion causes a scale-dependent shift in the relative reliability of spectral and spatial cues; repeated uniform aggregation dilutes subtle spectral signatures across levels, leading to degraded class separability among spectrally similar categories.
Simultaneously, target objects in aerial remote sensing and drone imagery exhibit arbitrary in-plane orientations, where rotation equivariance acts as an essential geometric inductive bias. Nevertheless, integrating rotation equivariance into multispectral detection reveals severe modeling conflicts: established equivariant architectures (such as G-CNNs and ReDet) rely strictly on discrete spatial sampling and cannot parameterize continuous spectral fields; complete equivariant designs like FRED introduce group-indexed deformable convolutions whose computational overhead scales quadratically; furthermore, conventional oriented detection heads duplicate parameters across discrete orientations, rendering networks excessively bulky.
Addressing these coupled dilemmas of spectral discreteness, scale-dependent fusion reliability, and computational inflation in equivariant modeling requires rethinking continuous representation learning on transformation groups. Core idea: factorize multispectral group features into rotation-equivariant spatial latent codes and Fourier-encoded continuous spectral coordinate fields for monotonic order-preserving spectral resampling, while adaptively gating cross-scale pyramid fusion via dual-prototype consistency weighting and predicting oriented boxes through a group-indexed parameter-shared head.
Method¶
Overall Architecture¶
The end-to-end pipeline of FressDet comprises four primary stages: group lifting, implicit continuous spectral resampling backbone, prototype-guided equivariant neck fusion, and group-indexed oriented detection head. An input multispectral image is first lifted to a group feature representation on discrete rotation group \(G = C_N\) (configured to \(C_4\) with \(90^\circ\) increments by default). Inside the SpeIWNet backbone, features are processed across four stages equipped with the Spectral Implicit Warp (SpeIW) module to adaptively resample continuous spectral signatures without violating channel order. The multi-scale group features are then routed into PAFPN, where the Rotation-Equivariant Consistency Weighting (ReCoW) module evaluates the position-wise reliability of spectral versus spatial branches to produce an adaptive convex combination. Finally, an Oriented-Aware Head decodes class-invariant probabilities and transformation-consistent oriented bounding boxes directly from group-indexed features without parameter duplication.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Multispectral Input Image<br/>H x W x C_in"] --> B["Group Lifting Layer<br/>Map planar input to rotation group G=C4"]
B --> C["Spectral Implicit Warp SpeIW<br/>Fourier-encoded coordinates + monotone cumulative resampling"]
C --> D["Rotation-Equivariant Consistency Weighting ReCoW<br/>Local window soft spectral & hard spatial routing with gating"]
D --> E["Oriented-Aware Head<br/>Group-invariant cls + cyclic-tied DFL + soft circular angle reg."]
E --> F["Oriented Bounding Box Predictions OBB<br/>(x, y, w, h, theta, class)"]
Key Designs¶
1. Spectral Implicit Warp (SpeIW): Continuous and Order-Preserving Spectral Resampling To overcome the limitations of discrete band processing, SpeIW applies the universal approximation property of implicit neural representations to the spectral axis. Each channel is assigned a normalized continuous coordinate \(\lambda_c \in [-1, 1]\) and mapped through an \(L\)-frequency Fourier positional encoding \(\psi(\lambda) = [\sin(2^i \pi \lambda), \cos(2^i \pi \lambda)]_{i=0}^{L-1} \in \mathbb{R}^{2L}\) to bypass the low-frequency spectral bias of coordinate networks. The warping field is factorized into an equivariant spatial latent code \(z(x, g) \in \mathbb{R}^R\) and a spectral basis decoded by an implicit MLP \(f_\theta\): $\(\delta_c(x, g) = \langle \mathbf{z}(x, g), f_\theta(\psi(\lambda_c)) \rangle, \quad c = 1, \dots, C\)$ Directly utilizing \(\delta_c\) as coordinates causes index crossing and breaks physical band ordering. SpeIW guarantees strict monotonicity by enforcing strictly positive increments via softplus \(\sigma_+\) followed by normalized cumulative summation: $\(\Delta_c(x, g) = \sigma_+(\delta_c(x, g)), \quad \phi_c(x, g) = (C - 1) \cdot \frac{\sum_{j=2}^c \Delta_j(x, g)}{\sum_{j=2}^C \Delta_j(x, g)}\)$ Here \(\phi_c(x, g)\) satisfies \(\phi_1 < \phi_2 < \dots < \phi_C\) within range \([0, C-1]\). Features are resampled via differentiable linear interpolation along the spectral axis. Because resampling is executed along the channel dimension independently per spatial-group coordinate \((x, g)\), it strictly commutes with spatial rotation, preserving full rotation equivariance.
2. Rotation-Equivariant Consistency Weighting (ReCoW): Dual-Prototype Routing with Adaptive Fusion Pyramid aggregation tends to blur discriminative spectral responses in deep layers while injecting spatial noise into shallow layers. ReCoW computes a group-averaged rotation-invariant representation \(\bar{F}(x) = \frac{1}{|G|}\sum_{g} F(x, g)\) within each \(W \times W\) window and extracts \(K = q^2\) (\(4 \times 4 = 16\)) region prototypes via adaptive pooling. Two complementary routing paths are tailored to spectral versus spatial characteristics: - Spectral Branch (Soft Assignment): Spectral signatures represent continuous mixed surface compositions. ReCoW computes temperature-scaled cosine similarities \(p_k(x) \propto \exp(\tau \cdot \cos(\mathbf{c}_k, \bar{F}(x)))\) to maintain spectral diversity, broadcasting the aggregated spectral feature \(\hat{F}_{\text{spec}}(x) = \sum_k p_k(x) \mathbf{c}_k\) identically across all group orientations. - Spatial Branch (Hard Assignment): Spatial structures demand sharp edge delineation. ReCoW generates scalar spatial prototypes \(\{\mu_k\}\) from intensity map \(s(x)\) and applies Gaussian affinity to perform crisp hard assignment \(k^*(x) = \arg\max_k \exp(-\gamma \|\mu_k - s(x)\|_2^2)\), gathering group features by cluster mean: $\(\hat{F}_{\text{spat}}(x, g) = \frac{1}{|\Omega_{k^*(x)}|} \sum_{x' \in \Omega_{k^*(x)}} F(x', g)\)$ A position-wise gating scalar \(\beta(x) = \sigma(\langle \pi_s(\bar{F}_{\text{spec}}(x)), \pi_t(\bar{F}_{\text{spat}}(x)) \rangle)\) measures subspace alignment between both branches to yield an adaptive convex combination \(F'(x, g) = \beta(x) F_{\text{spec}}(x, g) + (1-\beta(x)) F_{\text{spat}}(x, g)\), suppressing spurious background responses while maintaining equivariance.
3. Oriented-Aware Head: Group-Indexed Predictions without Parameter Replication To eliminate parameter redundancy and replication common in rotated bounding box predictors, the oriented-aware head takes neck group features \(F_n \in \mathbb{R}^{C \times |G| \times H \times W}\) and decouples symmetry constraints across task branches: - Classification Branch: Enforces orientation invariance via group mean pooling \(\mathbf{z}_{\text{inv}} = \frac{1}{|G|}\sum_{g \in G} \mathbf{z}_g\), eliminating orientation dependencies while maintaining spatial coordinate equivariance. - Bounding Box Regression Branch: Retains group-indexed features and implements a cyclic-tied Distribution Focal Loss (DFL) readout whose convolutional weights are tied to relative group offsets \(g^{-1}g'\). - Angle Regression Branch: Predicts group-indexed residual candidates decoded through soft circular aggregation under \(\pi\)-periodic wrapping: \(f_{\text{ang}}(\rho_g F_n)(x) = \text{wrap}_\pi(f_{\text{ang}}(F_n)(g^{-1}x) + \frac{2\pi}{|G|}\text{idx}(g))\). All branches leverage depthwise-pointwise group convolutions shared across all orientations, achieving compact, stable oriented predictions with an extremely small parameter footprint.
Loss & Training¶
FressDet is constructed upon the Ultralytics YOLO framework adapted for rotated object detection, expanding the stem convolution to intake variable multispectral band numbers. The model is trained from scratch without any rotation-based data augmentations, as internal rotation equivariance provides complete geometric inductive bias. The training objective aggregates classification cross-entropy loss \(\mathcal{L}_{\text{cls}}\), cyclic distribution focal loss \(\mathcal{L}_{\text{dfl}}\), and rotated IoU box regression loss \(\mathcal{L}_{\text{box}}\). Training was performed on dual NVIDIA RTX 3090 GPUs with PyTorch.
Key Experimental Results¶
Main Results¶
FressDet is benchmarked across five multispectral object detection datasets: MODA (primary benchmark), HOD3K, DroneVehicle, LLVIP, and VEDAI. FressDet establishes superior accuracy across aerial and ground perspectives while dramatically lowering parameter count and computational complexity.
| Dataset | Method | Backbone | Params (M) | FLOPs | \(\text{mAP}_{50}\) (%) | \(\text{mAP}_{75}\) (%) | mAP (%) |
|---|---|---|---|---|---|---|---|
| MODA | YOLO11m | CSP-D | 25.1 | 126.2G | 67.2 | 54.6 | 48.9 |
| MODA | ReDet | ReR-N-50 | 31.6 | 245.2G | 66.8 | 44.1 | 41.3 |
| MODA | MessDet | RE-CSP-N | 15.1 | 616.0G | 68.1 | 42.4 | 40.1 |
| MODA | OSSDet (Prev. SOTA) | ResNet-50 | 36.5 | 263.1G | 69.0 | 45.9 | 42.7 |
| MODA | FressDet (Ours) | SpeIWNet | 2.3 | 31.7G | 73.1 | 60.2 | 54.3 |
| HOD3K | Faster R-CNN | ResNet-50 | 41.1 | 206.7G | 89.4 | 62.7 | 56.9 |
| HOD3K | S2ADet | ResNet-50 | 48.6 | 169.2G | 93.4 | 66.2 | 59.8 |
| HOD3K | OSSDet (Prev. SOTA) | ResNet-50 | 36.6 | 131.2G | 93.4 | 68.8 | 60.9 |
| HOD3K | FressDet (Ours) | SpeIWNet | 2.3 | 35.2G | 93.8 | 69.1 | 61.1 |
| DroneVehicle | Dual-YOLO | CSP-D | - | - | 71.9 | - | 55.2 |
| DroneVehicle | WaveMamba (Prev. SOTA) | Mamba-based | - | - | 79.8 | - | 60.5 |
| DroneVehicle | FressDet (Ours) | SpeIWNet | 2.3 | 31.7G | 80.3 | - | 61.6 |
On challenging small-scale classes in MODA, FressDet demonstrates outstanding improvements: Bike AP climbs from 45.3% to 62.0% (+16.7%), and Pedestrian AP jumps from 45.7% to 72.5% (+26.8%) compared with OSSDet.
Ablation Study¶
Ablations on MODA validate the contribution of each architectural innovation and compare SpeIW against various spectral sampling operators.
Component-wise Ablation Study on MODA: | Equiv. Conv | SpeIW | ReCoW | Oriented Head | \(\text{mAP}_{50}\) (%) | \(\text{mAP}_{75}\) (%) | mAP (%) | Note | |:---:|:---:|:---:|:---:|:---:|:---:|:---:|---| | ✗ | ✗ | ✗ | ✗ | 62.0 | 48.0 | 43.3 | Non-equivariant baseline | | ✓ | ✗ | ✗ | ✗ | 66.8 | 54.1 | 47.7 | Equivariant convolutions alone deliver +4.4% mAP | | ✓ | ✓ | ✗ | ✗ | 69.9 | 55.8 | 50.3 | Adding SpeIW provides +2.6% mAP gain | | ✓ | ✗ | ✓ | ✗ | 69.0 | 55.1 | 49.6 | Adding ReCoW provides +1.9% mAP gain | | ✓ | ✗ | ✗ | ✓ | 68.6 | 55.5 | 49.3 | Adding OH provides +1.6% mAP gain | | ✓ | ✓ | ✓ | ✗ | 71.7 | 58.8 | 52.5 | Combining continuous spectral with pyramid fusion | | ✓ | ✓ | ✗ | ✓ | 71.6 | 58.4 | 52.2 | Combining continuous spectral with oriented head | | ✓ | ✓ | ✓ | ✓ | 73.1 | 60.2 | 54.3 | Full model (+11.0% mAP over baseline) |
Comparison of Spectral Operators in SpeIWNet Backbone: | Spectral Operator | \(\text{mAP}_{50}\) (%) | \(\text{mAP}_{75}\) (%) | mAP (%) | Mechanism Analysis | |---|---|---|---|---| | C4-CNN | 71.4 | 58.6 | 52.1 | Conventional discrete group convolution | | RE-DCN | 71.6 | 58.7 | 52.9 | Baseline rotation-equivariant deformable convolution | | RE-DCNv2 | 72.0 | 59.3 | 53.2 | Equivariant DCN with modulated scaling | | RE-DCNv3 | 71.7 | 59.0 | 53.1 | Multi-scale sparse grouping variant | | RE-DCNv4 | 71.9 | 59.2 | 53.0 | High-performance memory-optimized variant | | SpeIW (Ours) | 73.1 | 60.2 | 54.3 | Continuous coordinate field + monotonic order preservation |
Sensitivity and Design Choices: - Rotation Group Order: Moving from \(C_2\) (71.2% \(\text{mAP}_{50}\) / 1.1M params / 15.4G FLOPs) to \(C_4\) (73.1% \(\text{mAP}_{50}\) / 2.3M params / 31.7G FLOPs) yields significant gains; expanding further to \(C_8\) yields negligible improvement (73.2% \(\text{mAP}_{50}\), +0.1%) while doubling FLOPs to 65.1G, confirming \(C_4\) (\(90^\circ\)) as the optimal trade-off. - Fourier Frequency \(L\): Performance rises monotonically from \(L=0\) (71.5%) to peak at \(L=8\) (73.1%), with a slight decline at \(L=16\) (72.9%), demonstrating that high-frequency positional encoding is vital for resolving abrupt spectral transitions. - Prototype Routing Scheme: Soft spectral + hard spatial routing achieves the highest 73.1% \(\text{mAP}_{50}\), outperforming hard-hard (70.4%), soft-soft (71.6%), and hard-soft (69.8%), corroborating that routing biases must align with domain characteristics.
Key Findings¶
- Continuous Monotonicity Trumps Discrete Offsets: Deformable convolutions cannot constrain channel offsets to respect monotonic physical ordering, frequently triggering channel permutation. SpeIW uses implicit fields and softplus cumulative summation to maintain strict monotonic resampling, surpassing the strongest RE-DCN variant by 1.1% mAP.
- True Geometric Invariance without Data Augmentation: Without rotation augmentation, FressDet maintains steady performance under fine-grained \(1^\circ\)-interval rotational perturbations, whereas baseline architectures experience extreme variance across angles.
- Extreme Parameter Efficiency: FressDet achieves state-of-the-art detection accuracy with merely 2.3M parameters—a 93% reduction compared to OSSDet (36.5M)—proving that proper inductive biases outperform parameter brute-forcing.
Highlights & Insights¶
- Order-Preserving Coordinate Warping: SpeIW solves a critical flaw in implicit neural representation modeling for discrete multi-channel signals: preventing coordinate crossing by cumulating strictly positive activations.
- Heterogeneous Dual Prototype Routing: Designing distinct routing paradigms (soft assignment for smoothly varying spectral mixtures and hard assignment for sharp geometric edges) establishes an elegant blueprint for multimodal feature fusion.
- Lightweight Group-Indexed Readouts: Sharing parameters across group orientations and utilizing cyclic DFL indexing proves that complete rotation equivariance can be engineered with negligible parameter budgets.
Limitations & Future Work¶
- Discrete Group Approximation: The architecture is discretized to the cyclic subgroup \(C_4\); orientations outside multiples of \(90^\circ\) still depend on linear interpolation approximations, suggesting future research toward continuous Lie group steerable representations.
- Scaling to Hyperspectral Imagery (HSI): The model was evaluated on multispectral imagery with up to tens of bands; scaling to hundreds of narrow bands in hyperspectral remote sensing may require sparse implicit querying strategies.
Related Work & Insights¶
- vs ReDet / FRED: While ReDet provides approximate equivariance and FRED suffers from heavy group-indexed deformable computation, neither models continuous spectral physics. FressDet integrates continuous spectral implicit warping with strict equivariance, slashing parameter counts by over 90%.
- vs OSSDet / S2ADet: Prior multispectral detectors treat bands as discrete channels with standard attention, leaving deep features vulnerable to spectral washing. FressDet proves that continuous spectral resampling coupled with geometric symmetry provides a much stronger inductive foundation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Seamlessly unifies continuous implicit spectral resampling with rotation-equivariant group architectures.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across five benchmark datasets with comprehensive ablations, perturbation tests, and visualizations.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, clear motivation, and self-contained theoretical derivations.
- Value: ⭐⭐⭐⭐⭐ SOTA performance achieved with an ultralight 2.3M parameter footprint, ideal for deployment on edge devices and aerial drones.