SGP2: Coarse-to-Fine Controllable Multimodal Remote Sensing Image Generation¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/cpy0029/MMEarth-1.5M
Area: Image Generation
Keywords: remote sensing image generation, controllable multimodal generation, Grassmann manifold, physical prior alignment, spectral loss
TL;DR¶
To tackle semantic conflicts and physical inconsistencies in remote sensing image synthesis, SGP2 establishes a coarse-to-fine generation framework that disentangles global vs. detailed text concepts along Grassmann manifold geodesic trajectories and dynamically routes multi-level visual foundation features constrained by frequency-domain amplitude and phase spectral losses.
Background & Motivation¶
Diffusion-based text-to-image synthesis has established a dominant paradigm across natural image domains, creating unprecedented opportunities to mitigate data scarcity and fuel downstream Earth observation tasks such as land cover classification and object detection. However, directly transferring natural image diffusion models to the remote sensing (RS) domain encounters formidable domain adaptation barriers. Remote sensing imagery exhibits specialized overhead perspectives, complex spatial structural topologies, and rigorous physical interaction rules with electromagnetic radiation, including distinct optical spectral responses and synthetic aperture radar (SAR) microwave scattering characteristics. Although existing ControlNet-style controllable RS generation models incorporate spatial layout priors such as edge maps, sketches, or semantic segmentation masks, they remain severely crippled by two critical bottlenecks when handling complex textual descriptions.
The primary bottleneck is semantic conflict. Complex remote sensing scenes intrinsically pack multi-tiered semantic hierarchies, combining macroscopic layout concepts (e.g., an "industrial area" containing prominent "oil tanks") with localized microscopic details (such as "pipelines"). Standard pretrained text encoders project text prompts as static, entangled points in Euclidean space. During progressive iterative denoising, high-variance macroscopic concepts overwhelmingly dominate attentional capacity, overshadowing submerged fine-grained descriptions and leading to incomplete details or critical semantic omissions. The second bottleneck is physical inconsistency. RS images represent real-world physical responses with strict radiometric energy profiles and geometric topological boundaries. Data-driven diffusion models lacking physical prior injection and frequency-domain energy constraints frequently suffer from structural collapse or severe radiometric distortion across the denoising trajectory.
The core insight for addressing these issues lies in moving beyond stage-agnostic, uniform condition injection and synchronizing geometric and physical priors with the intrinsic coarse-to-fine progression of diffusion denoising. Core idea: project textual embeddings onto the Grassmann manifold to evolve smoothly along dynamic geodesic trajectories that resolve Euclidean entanglement, while dynamically routing multi-level visual foundation features via a timestep-aware MixRouter and enforcing joint frequency-domain amplitude and phase spectral losses for physics-faithful multimodal remote sensing generation.
Method¶
Overall Architecture¶
Built upon a Diffusion Transformer (DiT) backbone, SGP2 coordinates three core components aligned with the temporal evolution of diffusion denoising. At the text condition branch, the Grassmann Miner (GM) applies singular value decomposition to text embeddings and constructs dynamic geodesic trajectories on the Grassmann manifold across denoising timesteps, enabling a smooth attentional handover from global layout anchors to localized fine details. Concurrently, multimodal conditional latent states are concatenated and fed into the DiT backbone. During intermediate denoising, the Physical Prior Expert (PPE) leverages a timestep-aware MixRouter to dynamically distribute alignment intensity across both deep semantic layers and shallow geometric layers of a Visual Foundation Model (VFM, such as DINOv3). Finally, a frequency-aware Spectral Loss (\(\mathcal{L}_{sp}\)) enforces joint amplitude and phase constraints on Fourier-transformed representations, producing structurally coherent, radiometrically authentic optical and SAR remote sensing images.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Text + Multimodal Condition Latents"] --> B["Grassmann Miner<br/>SVD subspace decomposition & geodesic trajectory evolution"]
B --> C["DiT Denoising Backbone<br/>Multimodal iterative latent denoising"]
C --> D["Physical Prior Expert<br/>MixRouter dynamic timestep-guided multi-level VFM routing"]
D --> E["Spectral Loss<br/>FFT joint amplitude and phase constraint"]
E --> F["Output High-Fidelity RS Images<br/>Optical Image / SAR Image"]
Key Designs¶
1. Grassmann Miner: Manifold Geodesic Trajectory for Text Semantic Disentanglement and Temporal Handover Conventional text encoders treat textual conditions as fixed points in Euclidean space, where dominant global concepts and granular local descriptions suffer from severe feature entanglement, leaving fine details suppressed by macroscopic context. To isolate hierarchical semantics without violating geometric orthogonality, SGP2 first applies Singular Value Decomposition (SVD) to the text feature matrix \(Y \in \mathbb{R}^{L \times D}\): $\(Y = U S V^\top\)$ where the basis is truncated to the top-\(k\) dimensions \(U_k \in \mathbb{R}^{L \times k}\). Capitalizing on SVD energy concentration, the top-\(k_1\) vectors \(U_g \in \mathbb{R}^{L \times k_1}\) encapsulate high-variance semantic anchors (e.g., broad landscape layout) that govern early denoising, while the remaining orthogonal vectors \(U_d \in \mathbb{R}^{L \times (k - k_1)}\) isolate granular nuances. To avoid destroying orthogonality through naive Euclidean interpolation, SGP2 propagates semantic representations along minimal-length geodesic paths on the Grassmann manifold \(\mathcal{G}\). For each subspace \(U_i\) (\(i \in \{g, d\}\)), a continuous geodesic path parameterized by angle \(\theta_i\) is formulated as: $\(U_i(\theta_i) = U_i \cos(\theta_i) + \Delta_i \sin(\theta_i)\)$ where tangent vector \(\Delta_i\) is derived by projecting a timestep-conditioned vector \(Z_i = \text{MLP}(\tau) \cdot \text{AvgPool}(U_i)\) onto the orthogonal complement of \(U_i\). Timestep scheduling functions \(w_g(\tau) = \sin(\tau\pi/2)\) and \(w_d(\tau) = \cos(\tau\pi/2)\) scale geodesic angles \(\theta_g = \alpha_g w_g(\tau)\) and \(\theta_d = \alpha_d w_d(\tau)\), while diagonal operator \(\hat{S} = S \odot \Gamma(\tau)\) recalibrates singular values. Reconstructing \(\hat{Y} = U_{final} \hat{\Sigma} V^\top\) and injecting the geometric increment \(P = \hat{Y} - Y\) through a zero-initialized projection operator \(W_p\) (\(Y_{out} = Y + P W_p\)) ensures stable, uncorrupted feature integration.
2. Physical Prior Expert: Timestep-Aware Dynamic Multi-Level Visual Feature Soft Routing Existing representation alignment schemes (such as REPA) restrict alignment to a rigid single-point mapping of top-level VFM semantic outputs, failing to inject the low-level spatial and geometric priors indispensable for overhead RS structural fidelity. SGP2 introduces the Physical Prior Expert, constructing a dense, multi-level cooperative alignment between the feature pyramids of DiT and pretrained VFMs (DINOv3). The core MixRouter module conditions on the normalized timestep \(\tau = t/T\) to generate a soft routing allocation matrix: $\(W(\tau) = \text{Softmax}(\text{MLP}(\tau)) \in \mathbb{R}^{N_D \times N_V}\)$ where entry \(W^{(i,j)}(\tau)\) specifies the alignment weight of VFM layer \(j\) onto DiT layer \(i\). The multi-level objective is formulated as the weighted expectation over feature pair similarities: $\(\mathcal{L}_{align} = \sum_{i=1}^{N_D} \sum_{j=1}^{N_V} W^{(i,j)}(\tau) \cdot [1 - \text{sim}(x_D^{(i)}, x_V^{(j)})]\)$ During early denoising stages (large \(t\)), MixRouter allocates maximal weight to deep VFM layers (e.g., Layer 23) to establish global structural footprints; as denoising progresses toward late stages (small \(t\)), alignment smoothly shifts toward shallow VFM layers (e.g., Layer 1) to inject fine-grained geometric edges and physical boundary details.
3. Spectral Loss: Joint Amplitude and Phase Frequency Constraints for Electromagnetic Fidelity Remote sensing imagery represents the ground physical response to electromagnetic irradiation, requiring strict fidelity in frequency-domain energy profiles beyond spatial matching. SGP2 introduces a frequency-aware Spectral Loss \(\mathcal{L}_{sp}\) via 2D Fast Fourier Transform (FFT) on DiT features \(x_D\) and VFM features \(x_V\). A log-smoothed L1 loss anchors the macroscopic energy and radiometric distribution across amplitude spectra: $\(\mathcal{L}_{amp} = \left| \log(|\mathcal{F}(x_D)|) - \log(|\mathcal{F}(x_V)|) \right|\)$ To enforce rigorous topological constraints on high-frequency structural boundaries, absolute errors of real and imaginary complex components are normalized relative to VFM amplitude: $\(\mathcal{L}_{st} = \frac{|Re(\mathcal{F}(x_D)) - Re(\mathcal{F}(x_V))| + |Im(\mathcal{F}(x_D)) - Im(\mathcal{F}(x_V))|}{|\mathcal{F}(x_V)| + \epsilon}\)$ By independently penalizing real and imaginary discrepancies, this term implicitly enforces phase coherence across frequency channels. A dynamic schedule balances the two objectives: $\(\mathcal{L}_{sp} = w_s(\tau) \mathcal{L}_{amp} + (1 - w_s(\tau)) \mathcal{L}_{st}\)$ where \(w_s(\tau) = \sin(\pi\tau/2)\). Optimization smoothly shifts from global radiometric energy alignment (\(\mathcal{L}_{amp}\)) in early iterations to sharp phase and topological alignment (\(\mathcal{L}_{st}\)) in late iterations.
Loss & Training¶
The overall training objective combines three complementary terms: $\(\mathcal{L}_{total} = \mathcal{L}_{DiT} + \mathcal{L}_{align} + \mathcal{L}_{sp}\)$ where \(\mathcal{L}_{DiT}\) denotes the rectified flow denoising loss. Training follows a two-stage strategy: text-to-image pre-training for 2 epochs on 1,198,845 image-text samples from MMEarth-1.5M, followed by controllable multimodal fine-tuning for 5 epochs on 66,124 strictly co-registered optical and SAR pairs. The CAME-8bit optimizer is employed with learning rates of \(4 \times 10^{-5}\) for pre-training and \(1 \times 10^{-5}\) for fine-tuning under bf16 mixed precision, totaling roughly 85K optimization steps. At inference, Flow-DPM-Solver generates high-fidelity images in 20 denoising steps with a classifier-free guidance scale of 4.5.
Key Experimental Results¶
Main Results¶
On the RSICD test set for \(512 \times 512\) text-to-optical image generation, SGP2 substantially outperforms previous state-of-the-art methods across all metrics. For text-to-SAR generation on the DFC25 validation set, SGP2 also delivers top structural similarity and SAR-specific radiometric metrics.
| Dataset | Method | IS↑ | FID↓ | CLIPs↑ | OA(%)↑ |
|---|---|---|---|---|---|
| RSICD test set (Optical) | CRS-Diff (TGRS 2024) | 18.39 | 50.72 | 20.33 | 69.21 |
| RSICD test set (Optical) | InstancedDiff (CVPR 2024) | - | 138.60 | 24.70 | - |
| RSICD test set (Optical) | Text2Earth (GRSM 2025) | 18.17 | 53.15 | 22.62 | 68.27 |
| RSICD test set (Optical) | AeroGen(FT) (CVPR 2025) | 16.83 | 58.12 | 22.31 | 66.72 |
| RSICD test set (Optical) | JoDI (ArXiv 2025) | 4.28 | 181.36 | 20.23 | 22.74 |
| RSICD test set (Optical) | JoDI(FT) (ArXiv 2025) | 18.03 | 57.24 | 21.46 | 67.59 |
| RSICD test set (Optical) | SGP2 (Ours) | 21.23 | 44.17 | 24.58 | 86.27 |
On the DFC25 validation set for SAR synthesis, SGP2 achieves an SSIM of 0.569 (+10.9% over JoDI(FT)'s 0.513), a PSNR of 23.14 dB, an Equivalent Number of Looks (ENL) of 5.31, and reduces Radiometric Resolution (RaRes) to 1.76.
Ablation Study¶
On single-condition optical generation, ablation of the Grassmann Miner (GM), Physical Prior Expert (PPE), and Spectral Loss (\(\mathcal{L}_{sp}\)) demonstrates their individual and synergistic impacts on FID:
| Config (GM / PPE / \(\mathcal{L}_{sp}\)) | HED↓ | MLSD↓ | Dep.↓ | Skh.↓ | Seg.↓ | Road↓ | Avg. FID↓ | Note |
|---|---|---|---|---|---|---|---|---|
| Baseline (✗ / ✗ / ✗) | 64.7 | 66.5 | 65.1 | 60.6 | 64.7 | 69.3 | 65.15 | DiT baseline |
| Add GM only (✓ / ✗ / ✗) | 61.7 | 66.2 | 63.9 | 59.3 | 62.4 | 66.1 | 63.27 | Geodesic semantic disentanglement |
| Add PPE only (✗ / ✓ / ✗) | 55.8 | 57.4 | 56.2 | 56.8 | 60.9 | 61.3 | 58.07 | Multi-level VFM prior routing |
| Add PPE + \(\mathcal{L}_{sp}\) (✗ / ✓ / ✓) | 55.1 | 54.7 | 53.8 | 53.2 | 57.9 | 59.6 | 55.72 | Frequency amplitude and phase constraints |
| Add GM + PPE (✓ / ✓ / ✗) | 47.6 | 49.7 | 50.2 | 54.9 | 52.1 | 51.4 | 50.98 | Geometric and spatial feature alignment |
| Full SGP2 (✓ / ✓ / ✓) | 42.4 | 43.7 | 42.1 | 41.8 | 42.6 | 43.2 | 42.63 | Full synergistic integration |
Component variants within the Grassmann Miner (evaluated by average FID): - Full GM: 42.6 - Without SVD (w/o SVD): 45.8 (+3.2 degradation, confirming the necessity of orthogonal decomposition) - Without separating global and detail subspaces (w/o \(U_g\) & \(U_d\)): 44.3 (details overshadowed by global context) - Without geodesic trajectory (w/o geodesic trajectory): 43.6 (orthogonality violated by Euclidean operations) - Without dynamic spectral modulation (w/o \(\Gamma(\tau)\)): 43.1 (impeded energy centroid transfer)
Key Findings¶
- Synergistic Triad Integration: While PPE provides the largest single-module performance boost (reducing average FID from 65.15 to 58.07), coupling it with Spectral Loss and GM drives FID under sketch and depth conditions down by 31.0% and 35.3% relative to the baseline.
- Intuitive Semantic Disentanglement: Fixing the global subspace basis \(U_g\) maintains overarching scene layouts (e.g., farmland or athletics tracks), whereas modulating detail vectors \(U_d\) exclusively directs localized structures (river meanders or running lane textures), verifying successful elimination of semantic conflict.
- Physically Grounded Timestep Routing: Weight visualizations from MixRouter confirm that deep semantic layers (e.g., Layer 23) receive peak weights during early denoising steps, whereas shallow geometric layers (e.g., Layer 1) dominate late denoising steps, naturally aligning feature abstraction with physical generation dynamics.
Highlights & Insights¶
- Manifold Geometry Resolves Euclidean Semantic Entanglement: Rather than treating text embeddings as static points in Euclidean space, SGP2 models text representation paths along Grassmann geodesics, enabling seamless temporal migration of attention without distorting semantic orthogonality.
- Dynamic Soft Routing Overcomes Rigid Alignment: SGP2 abandons single-layer representation distillation in favor of a timestep-conditioned routing matrix, allowing the DiT backbone to dynamically draw upon multi-level geometric and semantic priors on demand.
- Complex Frequency Constraints Ensure Physical Coherence: Jointly penalizing Fourier real and imaginary errors enforces both amplitude energy matching and phase topological alignment, effectively eliminating structural collapse and radiometric artifacts in remote sensing generation.
Limitations & Future Work¶
- Computational Overhead in Manifold Projections: Although Grassmann tangent projections and multi-level VFM routing deliver superior fidelity, the additional SVD and FFT operations incur computational overhead during training.
- Ultra-High Resolution Scaling: The current architecture is optimized for \(512 \times 512\) tiles; synthesizing gigapixel-scale satellite panoramas requires investigating seamless cross-tile radiometric and phase harmonization under sliding-window or pyramidal inference.
Related Work & Insights¶
- vs CRS-Diff / Text2Earth: Traditional controllable RS models rely on rigid ControlNet structures or fixed prompt conditioning, frequently dropping textual details or distorting fine physical lines. SGP2 achieves significant quality gains (e.g., reducing RSICD FID from 50.72 to 44.17) through manifold-guided text evolution and frequency-domain regularization.
- vs REPA & Representation Alignment: Standard REPA aligns only the final latent feature of DINOv2 with DiT. In contrast, SGP2's Physical Prior Expert establishes a dynamic multi-level pyramid mapping spanning both deep semantic and shallow geometric features, reinforced by frequency-domain spectral losses.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant integration of Grassmann manifold geodesic geometry and frequency-domain phase constraints into remote sensing diffusion.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across optical and SAR modalities with single- and multi-condition controls, supported by the curated 1.5M MMEarth benchmark.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, systematically organized, and clearly motivated.
- Value: ⭐⭐⭐⭐⭐ Sets a compelling coarse-to-fine benchmark for controllable remote sensing synthesis and provides a valuable multimodal paired dataset.