M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detection¶

Conference: ECCV 2026
arXiv: 2505.10931
Code: https://github.com/wchao0601/M4-SAR
Area: Object Detection / Multi-Source Remote Sensing Fusion
Keywords: Optical-SAR fusion, oriented object detection, multi-source remote sensing dataset, Mamba cross-modal interaction, semi-supervised annotation

TL;DR¶

This paper proposes M4-SAR—the first large-scale optical-SAR paired oriented object detection dataset (112,174 image pairs, 981,862 instances, 6 categories), accompanied by a unified evaluation tool MSRODet and an end-to-end fusion detection framework E2E-OSDet (FAM + CMIM + AFM). After fusion, mAP improves by up to 5.1% compared to the best single-source modality, with particularly significant gains in complex environments.

Background & Motivation¶

B. Gap-filling Type

Optical remote sensing images are rich in texture but severely affected by illumination, clouds, fog, and low resolution. SAR images are all-weather and all-time but sensitive to speckle noise and characterized by sparse semantic information. The two are highly complementary—optical provides texture and color, while SAR provides geometric structure and robustness to adverse weather. Integrating both can significantly enhance the reliability of object detection in complex environments. However, this direction has long suffered from the lack of large-scale, high-quality, standardized optical-SAR paired datasets. Existing SAR datasets (HRSID, SSDD, MSAR, SARDet-100K, FAIR-CSAR, RSAR) are all single-source, and only OGSOD-1.0/2.0 provide optical-SAR pairs but are small in scale (about 20,000 pairs of \(256 \times 256\) images), contain few categories (3 classes), and their test sets only include SAR images, being designed for cross-modal knowledge distillation rather than end-to-end fusion detection evaluation.

This task is viable now due to two key enabling factors: (1) ESA's Sentinel-1/2 satellites provide globally covered, open, and free multi-resolution (10M/60M) optical and multi-polarization (VH/VV) SAR data; (2) the maturity of semi-supervised optical-assisted annotation strategies—leveraging the high semantic readability of optical images to crack the difficulty of high-cost SAR annotation. The design choice of this paper is: focus on 6 categories of static objects in coastal infrastructure scenes (bridges, harbors, oil tanks, playgrounds, airports, wind turbines), utilizing geographic coordinate alignment (rather than pixel-level registration) to reflect real cross-sensor acquisition conditions, and controlling the time interval within 10 days to ensure object structural consistency.

Core Idea: Construct the first large-scale optical-SAR paired detection benchmark using Sentinel public data and iterative semi-supervised optical-assisted annotation, and design the end-to-end fusion framework E2E-OSDet from three levels—data input (FAM), domain alignment (CMIM), and feature fusion (AFM)—to systematically narrow cross-modal domain discrepancies.

Method¶

Overall Architecture¶

This work consists of two parts: the M4-SAR dataset construction pipeline and the E2E-OSDet end-to-end fusion detection framework. Dataset construction consists of three stages—(a) Data acquisition: obtaining Sentinel-1 SAR (SLC format, VH/VV dual-polarization) and Sentinel-2 optical (RGB bands, L2A products, cloud coverage 0-10%) from the ESA open platform, covering coastal cities like Tianjin, Los Angeles, Hong Kong, and London, spanning 2020-2022; (b) Preprocessing and alignment: SAR undergoes multi-look average filtering for speckle reduction, filtering, DEM geocoding, and radiometric calibration, and optical undergoes band synthesis, is aligned with SAR based on geographic coordinates, is cropped into \(1024 \times 1024\) patches, and only valid overlap regions are retained; (c) Semi-supervised iterative annotation: first manually annotating 5% cloud-free optical images to train a YOLO detector, generating pseudo-labels which are manually corrected and added to the training set, and iterating until the entire dataset is annotated, eventually replacing with cloud-contained images and cropping into \(512 \times 512\) patches. The E2E-OSDet end-to-end detection framework projects SAR into a handcrafted feature space via FAM at the data input end to narrow the domain gap, implements cross-modal Mamba sequential interaction via CMIM at the feature level, enhances key region attention via AFM at the fusion layer, and outputs results via an oriented bounding box detection head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sentinel-1/2 Data Acquisition<br/>SAR SLC + Optical L2A"] --> B["Preprocessing & Geo-alignment<br/>Multi-look/Filtering/Calibration/Patching"]
    B --> C["Semi-supervised Iterative Annotation<br/>5% Manual→YOLO→Pseudo-labels→Correction"]
    C --> D["M4-SAR Dataset<br/>112K pairs, 982K instances, 6 classes"]
    D --> E["FAM: Filter Augment Module<br/>HOG/Canny/Haar/Grad/WST"]
    E --> F["CMIM: Cross-modal Mamba Interaction<br/>Interleaved Rearrangement + Multi-direction Scan"]
    F --> G["AFM: Area-Attention Fusion<br/>Stripe Weighting + Residual Connection"]
    G --> H["Oriented Bounding Box Detection Output<br/>6 classes"]

Key Designs¶

1. Semi-supervised optical-assisted iterative annotation: Leveraging high semantic readability of optical images to solve the high-cost SAR annotation bottleneck

Due to speckle noise and low contrast, it is difficult for non-experts to accurately determine object boundaries in SAR images; whereas, cloud-free optical images have clear boundaries and explicit semantics. The core insight of this paper is: in remote sensing scenarios, the cross-modal structural consistency of static infrastructure objects (bridges, harbors, etc.) is much higher than that of dynamic objects (ships); thus, coarse alignment based on geographic coordinates is sufficient to support reliable optical-to-SAR annotation transfer. The specific flow is: first, manually annotate all 6 categories of objects on 5% of cloud-free optical images to train a YOLO detector; then, use this detector to generate pseudo-labels for an additional 5% of data, manually correct them, add them to the training set, and retrain the detector; iterate this process until the entire dataset is annotated. During this process, images without valid objects are excluded to improve efficiency, and cloud-free images are eventually replaced with cloud-contained ones to reflect real-world scenarios. Ultimately, manual spot checks on 3,000 samples show an annotation accuracy exceeding 98% (98.3% for remote sensing experts, 99.3% for non-experts), and the minimal difference between experts and non-experts demonstrates that annotating on optical images does not require a remote sensing professional background. The annotation format is a four-point quadrilateral \([category, (x_1,y_1), (x_2,y_2), (x_3,y_3), (x_4,y_4)]\), with coordinates normalized to \([0,1]\).

2. FAM (Filter Augment Module): Projecting onto handcrafted features to narrow the optical-SAR pixel-domain gap

The differing imaging mechanisms of optical and SAR (optical relies on reflection, while SAR relies on microwave backscattering) lead to a significant distribution shift in the pixel domain (confirmed via feature visualization in Fig. 4a of the original paper), and direct fusion introduces noise. The core idea of FAM is: instead of altering the raw images, low-dimensional sparse SAR and optical representations are projected into a high-dimensional discriminative handcrafted feature space through classical image filters, where the cross-modal structural similarity (SSIM) is significantly higher than in the pixel domain. Given an optical image \(I_O \in \mathbb{R}^{H \times W \times 3}\) and a SAR image \(I_S \in \mathbb{R}^{H \times W \times 3}\), FAM generates augmented features for each filter \(t \in \{\text{WST, Canny, Haar, HOG, Grad}\}\):

\[F_O = I_O + \alpha * \mathbb{FAM}^t(I_O), \quad F_S = I_S + \alpha * \mathbb{FAM}^t(I_S)\]

where \(\mathbb{FAM}^t(\cdot)\) is the filter transformation (Canny edge detection, HOG, Haar-like wavelet features, Grad edge detection based on ratio of average gradients, WST wavelet scattering transform), and \(\alpha\) is a learnable scalar weight. Optical and SAR features each go through their own branch, injecting noise-robust multi-scale edge and shape cues while preserving their respective modal-specific structures. Experiments show that the Grad feature achieves the best performance—because its gradient ratio operator renders it naturally robust to multiplicative SAR speckle noise, and it can capture multi-scale gradient information. Note that FAM is inserted at the data input end with near-zero parameter overhead (only the scalar \(\alpha\)), adding no inference burden, making it an extremely efficient domain adaptation approach.

3. CMIM (Cross-modal Mamba Interaction Module): Interleaved rearrangement to achieve fine-grained cross-modal sequence interaction

Existing fusion methods mainly focus on spatial-level feature concatenation (concat/add), ignoring distribution biases in the feature domain. Mamba's state space modeling excels at capturing long-range dependencies, but existing Mamba-based fusion architectures are complex with bloated interaction mechanisms. The key innovation of CMIM lies in Interleaved Input Rearrangement (IIR): traditional methods directly concatenate optical and SAR feature sequences end-to-end \(Z^{tra}_j = [p^o_1, ..., p^o_n, p^s_1, ..., p^s_n]\), but such coarse-grained concatenation fails to establish position-by-position cross-modal correspondence. IIR interleaves the features of both modalities patch-by-patch:

\[Z^{IIR}_j = [p^o_1, p^s_1, p^o_2, p^s_2, \dots, p^o_n, p^s_n]\]

Each optical patch is adjacent to its spatially corresponding SAR patch in the sequence, allowing Mamba's state space modeling to directly capture cross-modal semantic correspondences within short time-steps. The interleaved sequence is then fed into a directional scanning mechanism \(Z^{scan}_j = \mathbb{SCAN}^m(Z^{IIR}_j)\) (supporting four scanning methods: Bidirectional, Z-Order, Zigzag, and Hilbert). After modeling along horizontal and vertical directions via Mamba blocks, the sequence is reshaped back to 2D space, with original modal information preserved via residual connections. CMIM operates independently on three feature layers (\(j=3,4,5\), corresponding to 8/16/32x downsampling): \(\{F'_{Oj}, F'_{Sj}\} = \mathbb{CMIM}(Z^{scan}_j)\). The deep mechanism of IIR is to convert "cross-modal alignment" into "local adjacency relationships within the Mamba sequence," allowing synchronous semantic enhancement to occur within a short time-step, effectively preventing feature drift and gradient inconsistency during training. Experiments demonstrate that CMIM exhibits strong robustness to different scanning strategies.

4. AFM (Area-Attention Fusion Module): Stripe area-attention for lightweight and efficient feature fusion

After FAM preprocessing and CMIM cross-modal interaction, the features of both modalities need to be integrated into a unified detection representation at the fusion layer. Inspired by the efficient spatial modeling of YOLOv12, AFM crops the \(H \times W\) feature map along one direction into \(k\) non-overlapping stripe blocks (sizes \(H/k \times W\) or \(H \times W/k\)), achieving block division via a simple reshape without explicit window partitioning. Within each block, attention weights are computed to adaptively weigh the optical and SAR features:

\[F^{fus}_j = \mathbb{AFM}(F'_{Oj} + F_{Oj}, F'_{Sj} + F_{Sj}), \quad j \in \{3,4,5\}\]

where \(F'_{Oj}, F'_{Sj}\) are the CMIM-enhanced features, and \(F_{Oj}, F_{Sj}\) are the residual raw features. The advantages of AFM lie in three aspects: (1) block-level operations significantly reduce computational cost compared to global attention; (2) area-attention adaptively highlights key target regions and suppresses background clutter instead of relying on simple element-wise addition; (3) residual connections preserve raw modal information, preventing over-fusion. In ablation experiments, AFM alone contributes the most (+2.0 mAP), verifying the critical value of area-aware fusion in multi-source detection.

A Complete Example: Forward Propagation of an Optical-SAR Image Pair in a Harbor Scenario¶

Tracing the complete forward propagation of E2E-OSDet using a harbor scene as an example. Input: optical image \(I_O\) (512x512x3, 10M resolution, RGB three-channel, containing harbor docks, ship berthing areas, and breakwaters) and paired SAR image \(I_S\) (512x512x3, VH polarization, obvious speckle noise, harbor outlines visible but details blurry). First, FAM applies Grad filter enhancement to both images: on the optical side, the Grad operator extracts building edges and coastline gradients; on the SAR side, Grad suppresses speckle noise through gradient ratio calculation and extracts structural outlines, yielding \(F_O\) and \(F_S\) (each 512x512x3). Second, the YOLOv11-S backbone extracts multi-scale features layer-by-layer: optical features \(F_{Oj}\) and SAR features \(F_{Sj}\) are obtained at the 3rd layer (128x128xC3), 4th layer (64x64xC4), and 5th layer (32x32xC5). Third, at each scale, CMIM flattens \(F_{Oj}\) and \(F_{Sj}\) into 1D sequences and interleaves them using IIR (e.g., \(p^o_1, p^s_1, p^o_2, p^s_2, ...\)). The sequence is passed into Mamba blocks via Hilbert directional scanning for long-range interaction, outputting enhanced features \(F'_{Oj}, F'_{Sj}\). Fourth, AFM adds residuals to the enhanced features of each layer and performs area-attention fusion block-by-block, yielding \(F^{fus}_j\), which is then fed into the YOLOv8 OBB detection head. Final output: the harbor area is predicted as a rotated bbox (AP50=91.8%), along with its orientation angle and confidence score. The entire process contains 24.7M parameters with an inference time of 20.9ms.

Loss & Training¶

E2E-OSDet adopts a composite loss to jointly optimize classification and rotated bounding box localization:

\[\mathcal{L}_{total} = \mathcal{L}_{reg} + \mathcal{L}_{dfl} + \mathcal{L}_{cls}\]

where \(\mathcal{L}_{reg}\) is the ProbIoU rotated bounding box regression loss (modeling the rotated box as a 2D Gaussian distribution and calculating the Bhattacharyya distance), \(\mathcal{L}_{dfl}\) is the Distribution Focal Loss (which applies weighted focus to the discrete probability of the bounding box distribution), and \(\mathcal{L}_{cls}\) is the binary cross-entropy classification loss. All fusion models uniformly employ the YOLOv11-S backbone and YOLOv8 OBB detection head, optimized using SGD (lr=0.01, momentum=0.937, weight decay=5e-4) for 300 epochs, with a batch size of 64 and an input resolution of 512x512. Training is performed on dual RTX 4090 GPUs using FP16 mixed precision, and all fusion models are trained from scratch without loading pre-trained weights. The dataset is split into train:val:test = 5:2:3. In the MSRODet evaluation tool, single-source MMRotate methods (FCOS/ATSS/RepPoint/RTMDet/PSC/LSKNet) use a ResNet-50 backbone + ImageNet pre-training and are trained with AdamW (lr=2e-4) for 72 epochs.

Key Experimental Results¶

Main Results¶

Method	Modality	mAP	AP50	AP75	Params (M)	Inference (ms)
YOLO-Master	Optical	56.3	80.0	63.9	29.3	15.7
YOLO-Master	SAR	48.5	73.9	53.1	29.3	15.7
CFT	O+S	59.9	84.6	68.9	53.8	40.6
CLANet	O+S	59.6	84.6	68.5	48.2	29.1
MMIDet	O+S	59.8	84.8	68.6	53.8	41.9
ICAFusion	O+S	58.8	84.5	67.3	29.0	23.6
CSSA	O+S	58.0	83.4	66.4	13.5	12.3
E2E-OSDet	O+S	61.4	85.7	70.3	24.7	20.9

Fusion improves mAP by +5.1 compared to the best optical-only baseline and by +12.9 compared to the best SAR-only baseline. E2E-OSDet's parameter count (24.7M) is nearly half that of the runner-up CLANet (48.2M), and its inference speed (20.9ms) ranks second among all fusion methods, achieving the best overall efficiency.

Ablation Study¶

Configuration	FAM	CMIM	AFM	Params	mAP	AP50	AP75
Baseline (CSSA)				14.17M	58.0	83.4	66.4
+FAM	Y			14.17M(+0M)	58.9	84.5	67.1
+CMIM		Y		19.01M(+5.68M)	59.6	84.9	68.6
+AFM			Y	19.84M(+4.84M)	60.0	84.6	68.9
+FAM+CMIM	Y	Y		19.84M(+4.84M)	61.0	85.5	69.9
+FAM+AFM	Y		Y	19.01M(+5.68M)	60.7	84.9	69.5
+CMIM+AFM		Y	Y	24.69M(+10.52M)	60.9	85.2	70.0
Full	Y	Y	Y	24.69M(+10.52M)	61.4	85.7	70.3

Cross-Dataset Generalization¶

Method	OGSOD-1.0 mAP	OGSOD-2.0 mAP
YOLOv11 (Optical)	61.5	58.5
YOLOv11 (SAR)	47.8	45.9
CFT (O+S)	65.3	61.4
ICAFusion (O+S)	66.1	62.2
MMIDet (O+S)	66.7	63.0
E2E-OSDet (O+S)	67.3	65.1

Key Findings¶

AFM contributes the most: Adding AFM alone brings +2.0 mAP, indicating that area-aware fusion is far more effective than simple element-wise fusion; FAM alone contributes the least (+0.9 mAP), but its zero-parameter growth provides the highest cost-effectiveness.
Fusion is far superior to direct data mixing: Direct mixing of optical and SAR data to train a single-source detector actually leads to decreased performance (YOLO26 with mixed data achieves only 47.3% mAP, lower than optical at 55.4% and SAR at 48.2%), proving that cross-modal domain discrepancies must be processed through specialized fusion architectures, as simple mixing only introduces interference.
Grad features perform best among the five handcrafted features: Grad is naturally robust to multiplicative SAR speckle noise through gradient ratio operations, and has the highest cross-modal SSIM; other filters (Canny/HOG/Haar/WST) also yield positive gains, showing that "noise pattern matching" rather than "matching after noise reduction" is the correct approach to narrow the domain gap.
Insensitive to hyperparameters: Changes in the handcrafted feature input ratio and AFM area block size affect final performance by less than 0.5 mAP, demonstrating excellent deployment stability of the framework.

Highlights & Insights¶

Pragmatic design philosophy of the "four-multi" dataset: It does not force pixel-level registration (which is unobtainable in reality) but embraces the constraints of coarse coordination alignment, focusing on static infrastructure objects to bypass temporal misalignment. This approach of "acknowledging limitations and doing the best within constraints" is highly referenceable for benchmark papers.
Clever "zero-parameter domain adaptation" of FAM: Instead of adding learnable parameters, it narrows domain discrepancies solely by projecting onto classic handcrafted features—essentially utilizing human prior knowledge (edges, gradients, textures) to perform feature space transformations, allowing the network to learn fusion in a space that is easier to align. This highlights that injecting handcrafted feature priors in multi-modal fusion remains a cost-effective way to close the domain gap.
CMIM's IIR interleaved rearrangement is a universal design pattern: Converting "cross-modal alignment" into "local adjacency within Mamba sequences" is a trick that can be applied to any Mamba-based fusion task requiring the alignment of two-stream features (e.g., RGB-IR, RGB-Depth, RGB-Event), not restricted to remote sensing.
AFM's stripe-attention is highly transferable: The concept is directly applicable to other dual-stream fusion detection tasks, with computational costs much lower than global cross-attention, and without depending on predefined windows—representing a natural generalization of YOLOv12's area-attention to cross-modal scenarios.
Annotation accuracy >98% with expert/non-expert difference <1%: This indicates that annotating static infrastructure on optical images does not require a remote sensing professional background, which serves as a valuable reference for the feasibility and cost control of future dataset expansion.

Limitations & Future Work¶

Coarse geographic coordinate alignment limits application scope: Optical-SAR is aligned only via geographic coordinates (non-pixel-level registration), resulting in extremely low cross-modal similarity metrics (MSE 0.0662, SSIM 0.1256, MI 0.1729), rendering it unusable for tasks requiring precise alignment, such as image registration, change detection, or pixel-level fusion. The authors acknowledge this as a design trade-off—more closely reflecting real-world conditions at the cost of precise registration.
Limited and unbalanced category coverage: Only 6 classes of coastal infrastructure are included, with bridges accounting for 43.64% of instances while harbors account for only 2.83% (due to their large spatial coverage, where a single harbor is logged as one instance). It lacks dynamic classes like vehicles and ships, as well as diverse inland scenes such as mountains and farmland. Future work can introduce more fine-grained or dynamic categories.
Temporal alignment issue not deeply resolved: Sentinel-1/2 non-synchronous acquisition (interval < 10 days) excludes dynamic objects due to alignment failure. Future work could introduce temporal alignment strategies (multi-temporal modeling, optical flow warping) to extend coverage.
E2E-OSDet only validated on the YOLO framework: Although validated across YOLO versions (v5-v12) to show scalability, it has not been tested on DETR-like or anchor-free detectors, meaning module transferability requires further validation. The authors suggest designing lightweight fusion architectures to support edge deployment.

Compared with SARDet-100K / FAIR-CSAR / RSAR: These large-scale SAR datasets remain single-source and cannot support optical-SAR fusion research. M4-SAR fills this gap with paired data, showing an instance density (8.75/img) much higher than SARDet-100K (2.11/img), with denser scenes and higher detection difficulty.
Compared with OGSOD-1.0/2.0: These only contain 3 classes and ~20K pairs of \(256 \times 256\) low-resolution images, with test sets consisting only of SAR images, designed primarily for cross-modal knowledge distillation. M4-SAR comprehensively outperforms them in scale (5x image pairs, 18x instances), categories (2x), resolution (\(512 \times 512\)), and evaluation design (including both optical and SAR test sets).
Compared with RGB-IR fusion methods (CLANet/ICAFusion/CMADet/CSSA): The optical-SAR domain discrepancy (caused by completely different physical imaging principles) is much larger than in RGB-IR (which only has spectral band differences), making direct migration yields limited. This paper adapts 6 representative RGB-IR methods to the optical-SAR scenario and builds the MSRODet unified evaluation. It reveals that alignment-aware methods (CMADet) are more robust than direct fusion ones (ICAFusion), showing that alignment strategy is the primary factor in cross-modal fusion.
Compared with Mamba-based fusion methods like MambaSOD / FusionMamba: These methods feature highly complex interaction mechanisms. CMIM achieves highly efficient cross-modal interaction through the extremely simple IIR interleaved rearrangement, offering better control over parameter and computational overhead and remaining insensitive to scanning strategies.

Rating¶

Novelty: ⭐⭐⭐⭐ — The first large-scale optical-SAR paired oriented detection dataset and a tailored end-to-end fusion framework, filling a clear gap; however, the sub-modules (handcrafted feature enhancement, Mamba interaction, area-attention) are built on known techniques, making it a combinatorial innovation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ — Extremely comprehensive, covering 15+ single-source methods, 7 fusion methods, a unified MSRODet framework, ablation of 3 modules, comparisons of handcrafted feature types and scanning strategies, OGSOD cross-dataset generalization, mixed data experiments, and manual annotation quality assessment (8 annotators).
Writing Quality: ⭐⭐⭐⭐ — Well-structured with rich illustrations (10+ figures) and a detailed appendix; however, some technical details (such as the selection criteria for FAM's five filters and the ablation logic for CMIM scanning directions) could be explained more thoroughly.
Value: ⭐⭐⭐⭐⭐ — The M4-SAR dataset and MSRODet tool will have a long-term impact on the remote sensing multi-source fusion community, filling the infrastructure gap of "high demand, no data"; E2E-OSDet serves as the first dedicated baseline, providing a fair reference point for subsequent methods.