LGD-Net: Leader-Guided Cross-Modal Dynamics for Hyperspectral and Panchromatic Image Fusion

Conference: ECCV 2026
Paper: ECCV Official
Area: Remote Sensing
Keywords: Hyperspectral image fusion, continuous dynamical systems, cross-modal dynamics, graph propagation, Neural ODEs

TL;DR

LGD-Net proposes a hyperspectral and panchromatic fusion framework that explicitly decouples global cross-modal interaction from local multi-scale reconstruction, leveraging a continuous dynamical system on compact leader representations and discrepancy-aware graph propagation to achieve high-fidelity spectral and spatial reconstruction without increasing architectural depth.

Background & Motivation

Hyperspectral imaging captures detailed continuous spectral signatures across hundreds of contiguous narrow bands, providing valuable diagnostic information for critical earth observation tasks including environmental monitoring, mineral resource exploration, and precision agriculture. However, constrained by physical optical apertures and fundamental signal-to-noise ratio trade-offs, hyperspectral satellite sensors inevitably provide low spatial resolution (LRHS). Hyperspectral image fusion (pansharpening) aims to overcome this physical bottleneck by synthesizing high-resolution hyperspectral (HRHS) images from co-registered LRHS observations and complementary high-resolution panchromatic (PAN) images, simultaneously preserving rich spectral curves and enhancing fine spatial textures.

In recent years, deep learning approaches spanning convolutional neural networks, vision transformers, and hybrid architectures have advanced the state of the art in hyperspectral fusion. Nonetheless, conventional deep architectures overwhelmingly rely on stacked multi-layer designs, in which local spatial feature refinement and global cross-modal information exchange are tightly coupled and jointly updated across every layer of the network. Under this stacked paradigm, the propagation and interaction depth of panchromatic guidance are implicitly tied to the overall network depth. Consequently, strengthening global cross-modal interactions typically requires stacking deeper layers or inserting heavy attention blocks, which simultaneously intensifies local transformations and frequently triggers spatial over-smoothing or spectral distortion in homogeneous land-cover regions. Network depth functions as a single monolithic parameter controlling both global interaction and local reconstruction, restricting flexibility when balancing spatial sharpness against spectral fidelity.

To overcome this fundamental limitation, LGD-Net takes a radically different angle of attack: breaking the rigid coupling between global interaction and architectural depth. Instead of driving cross-modal interactions through deeper feed-forward layers, the framework summarizes multi-scale features from each modality into a compact global "leader" representation. Only this leader state undergoes a continuous-time dynamical evolution process (Neural ODE) across integration steps to capture long-range cross-modal context, while local feature nodes remain dedicated to fine spatial reconstruction. The refined leader is subsequently injected back into the local multi-scale hierarchy via residual modulation. Core idea: explicitly decouple global cross-modal interaction from local multi-scale reconstruction by formulating continuous dynamical state evolution on compact modality leaders and modulating multi-scale graph nodes for controlled, highly efficient hyperspectral pansharpening.

Method

Overall Architecture

Given a low-resolution hyperspectral input \(\mathbf{X}_{\text{HS}} \in \mathbb{R}^{C_{\text{in}} \times h \times w}\) and a high-resolution panchromatic input \(\mathbf{X}_{\text{PAN}} \in \mathbb{R}^{1 \times H \times W}\) with spatial resolution ratio \(r = H/h = W/w\), LGD-Net reconstructs a high-resolution hyperspectral image \(\widehat{\mathbf{X}} \in \mathbb{R}^{C_{\text{in}} \times H \times W}\). The full framework proceeds through four sequential stages: gated feature pyramid encoding, discrepancy-aware intra-modal graph propagation, continuous cross-modal dynamical evolution on compact leaders, and leader-guided residual modulation coupled with coordinate-conditioned decoding.

The overall data flow and component coordination are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: LRHS and PAN Images"] --> B["Gated Feature Pyramid Encoding<br/>Dilated depthwise convs and bidirectional multiscale hierarchy"]
    B --> C["Discrepancy-Aware Graph Propagation<br/>Adjacent-scale residual gating and structural alignment"]
    C --> D["Continuous Cross-Modal Dynamics<br/>Bilinear interaction and RK2 state evolution on leaders"]
    D --> E["Leader-Guided Refinement & Decoding<br/>Multiscale residual injection and coordinate-conditioned reconstruction"]
    E --> F["Output: Reconstructed HRHS Image"]

Key Designs

1. Discrepancy-Aware Graph Propagation: Intra-modal structural alignment via adjacent-scale residual gating

Following multiscale feature extraction via gated depthwise-dilated convolutions across pyramid levels, features at different scales remain hierarchically isolated. Naive multi-scale fusion or simple concatenation risks blurring sharp structural boundaries or diluting coarse-level contextual semantics. LGD-Net treats multi-scale feature maps as graph nodes and formulates inter-scale communication through discrepancy-aware residual updates within each modality.

Specifically, for each modality \(m \in \{\text{HS}, \text{PAN}\}\), multi-scale nodes are spatially aligned to PAN resolution via bilinear interpolation. The interaction between adjacent scales \(s\) and \(s-1\) (\(s > 1\)) is dynamically regulated by a channel-wise adaptive gating function \(\mathbf{g}_m^{(s)}\):

\[\widetilde{\mathbf{f}}_m^{(s)} = \mathbf{f}_m^{(s)} + \mathbf{g}_m^{(s)} \odot \phi(\mathbf{f}_m^{(s-1)})\]

where \(\phi(\cdot)\) is a learnable \(1 \times 1\) linear projection. The gating weight \(\mathbf{g}_m^{(s)}\) measures local structural divergence between adjacent scales:

\[\mathbf{g}_m^{(s)} = \sigma\left(\mathcal{N}\left(\psi(\mathbf{f}_m^{(s)} - \mathbf{f}_m^{(s-1)})\right)\right)\]

where \(\psi(\cdot)\) denotes depthwise spatial filtering followed by pointwise projection, \(\mathcal{N}(\cdot)\) denotes channel-wise Group Normalization, and \(\sigma(\cdot)\) is the Sigmoid activation. When structural discrepancy between adjacent scales is negligible, the gate converges toward a steady equilibrium, preventing over-correction across uniform regions; when substantial structural variance is detected along texture edges, the gate dynamically amplifies residual propagation, enforcing structural coherence while preserving localized spatial details.

2. Continuous Cross-Modal Dynamics: Decoupling interaction depth via leader state ODE evolution

Standard stacked architectures require deeper layer cascades to broaden the receptive field and deepen cross-modal interactions, needlessly inflating computation for local features. LGD-Net decouples this process by aggregating intra-modally refined multi-scale nodes into a single, compact global leader representation \(\mathbf{L}_m \in \mathbb{R}^{C \times H \times W}\) for each modality \(m \in \{\text{HS}, \text{PAN}\}\). The multi-scale local nodes are held fixed while only the two compact leaders evolve through a continuous dynamical system.

To capture both additive and multiplicative correlations across modalities, the framework first constructs bilinear cross-modal interaction terms:

\[\begin{cases} \mathbf{Z}_{\text{HS}}^{(t)} = \mathbf{L}_{\text{PAN}}^{(t)} + \mathbf{L}_{\text{HS}}^{(t)} \odot \mathbf{L}_{\text{PAN}}^{(t)} + \mathbf{D}_{\text{PAN}}(\mathbf{L}_{\text{PAN}}^{(t)}) \\ \mathbf{Z}_{\text{PAN}}^{(t)} = \mathbf{L}_{\text{HS}}^{(t)} + \mathbf{L}_{\text{HS}}^{(t)} \odot \mathbf{L}_{\text{PAN}}^{(t)} + \mathbf{D}_{\text{HS}}(\mathbf{L}_{\text{HS}}^{(t)}) \end{cases}\]

where \(\mathbf{D}_m(\cdot)\) denotes depthwise spatial filtering. These interaction states are projected into velocity fields \(\mathbf{F}_m^{(t)}\) via dynamic linear projections and modulated by a channel-shared spatial gate \(\boldsymbol{\gamma}^{(t)} = \sigma(\mathcal{G}([\mathbf{L}_{\text{HS}}^{(t)}, \mathbf{L}_{\text{PAN}}^{(t)}]))\). The governing differential equations are:

\[\frac{d\mathbf{L}_m}{dt} = \boldsymbol{\gamma}^{(t)} \odot \mathbf{F}_m^{(t)}\]

The continuous dynamics are solved using a second-order Runge-Kutta (RK2) numerical integration scheme with an adaptive effective step size \(\eta = \Delta t \cdot s\), where \(s \in (0, s_{\max})\) is a bounded learnable scalar. With just \(T=2\) discrete integration steps, the leader representations achieve deep cross-modal convergence without the excessive computational overhead of evolving high-dimensional local feature hierarchies.

3. Leader-Guided Refinement and Decoding: Global feedback injection with coordinate-conditioned reconstruction

After \(T\) continuous integration steps, the evolved hyperspectral leader \(\mathbf{L}_{\text{HS}}^{(T)}\) embodies globally aligned cross-modal context. This state is fed back into the local hyperspectral feature hierarchy to modulate each scale:

\[\widehat{\mathbf{f}}_{\text{HS}}^{(s)} = \widetilde{\mathbf{f}}_{\text{HS}}^{(s)} + \boldsymbol{\beta} \odot \mathcal{R}(\mathbf{L}_{\text{HS}}^{(T)})\]

where \(\mathcal{R}(\cdot)\) denotes a \(3 \times 3\) linear projection and \(\boldsymbol{\beta} \in (0, 1)^{1 \times H \times W}\) is a bounded spatial gating map derived from the normalized leader, ensuring stable numerical magnitude across layers.

The modulated multi-scale features are aggregated into a final unified feature \(\mathbf{L}_{\text{HS}}^\star\). Because standard convolutional decoders are translation-invariant and lack explicit spatial positional awareness, a normalized 2D Cartesian coordinate grid \(\mathbf{C} \in [-1, 1]^{2 \times H \times W}\) is concatenated along the channel dimension: \(\widetilde{\mathbf{L}}_{\text{HS}} = [\mathbf{L}_{\text{HS}}^\star, \mathbf{C}]\). A lightweight convolutional decoder then maps this coordinate-conditioned representation directly to the target HRHS image, achieving crisp structural reconstruction with minimal spectral distortion.

Loss & Training

LGD-Net is trained end-to-end using the standard \(L_1\) pixel-wise reconstruction loss between the predicted high-resolution hyperspectral output \(\widehat{\mathbf{X}}\) and the ground-truth reference \(\mathbf{X}_{\text{ref}}\):

\[\mathcal{L} = \|\mathbf{X}_{\text{ref}} - \widehat{\mathbf{X}}\|_1\]

The network is optimized using Adam for 1600 epochs. The initial learning rate is set to \(5 \times 10^{-4}\) and decayed to \(1 \times 10^{-5}\) via a cosine annealing schedule. Batch sizes are set to 8 for Botswana and Pavia Center, and 4 for Chikusei. Optimal solver parameters are established at \(T=2\) RK2 integration steps with a step scaling factor \(s=1.2\).

Key Experimental Results

Main Results

Experiments follow Wald's synthesis protocol on three widely benchmarked datasets: Botswana, Chikusei, and Pavia Center. Evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Spectral Angle Mapper (SAM), Cross-Correlation (CC), Root Mean Square Error (RMSE), and ERGAS.

Dataset	Method	PSNR ↑	SSIM ↑	SAM ↓	CC ↑	RMSE ↓	ERGAS ↓
Botswana	HyperPNN	37.430	0.944	2.454	0.969	0.019	1.544
	FPFNet	37.750	0.957	3.520	0.977	0.018	1.481
	HyperDSNet	38.010	0.947	2.308	0.972	0.018	1.421
	HspeNet	38.150	0.947	2.214	0.972	0.018	1.402
	HyperRefiner	39.200	0.959	2.094	0.978	0.015	1.249
	Lformer	39.210	0.957	2.087	0.978	0.015	1.258
	DHP-Darn	40.150	0.965	1.906	0.981	0.013	1.141
	MCIFNet	39.237	0.958	2.068	0.977	0.015	1.259
	LGD-Net (Ours)	41.856	0.979	1.695	0.987	0.011	1.016
Chikusei	HyperPNN	42.890	0.974	5.160	0.981	0.008	3.480
	FPFNet	41.030	0.969	5.589	0.977	0.010	3.947
	HyperDSNet	43.020	0.974	5.099	0.982	0.008	3.403
	HspeNet	42.840	0.975	4.917	0.982	0.008	3.401
	HyperRefiner	43.560	0.978	4.862	0.983	0.008	3.339
	Lformer	44.890	0.983	3.635	0.987	0.007	2.862
	DHP-Darn	45.400	0.985	4.431	0.988	0.006	2.724
	MCIFNet	45.101	0.984	4.514	0.988	0.007	2.814
	LGD-Net (Ours)	47.814	0.991	3.996	0.992	0.005	2.240
Pavia	HyperPNN	39.060	0.967	4.612	0.986	0.012	2.459
	FPFNet	39.320	0.966	6.115	0.986	0.013	2.451
	HyperDSNet	39.110	0.967	4.675	0.986	0.012	2.446
	HspeNet	38.370	0.965	4.382	0.984	0.013	2.625
	HyperRefiner	40.090	0.973	4.213	0.987	0.011	2.318
	Lformer	40.730	0.975	3.979	0.988	0.010	2.182
	DHP-Darn	41.670	0.978	3.820	0.990	0.009	2.018
	MCIFNet	41.365	0.976	3.903	0.989	0.010	2.070
	LGD-Net (Ours)	44.830	0.984	3.297	0.992	0.007	1.693

Ablation Study

Component-wise ablations isolate the contribution of each proposed architectural module:

Config	PSNR ↑	SSIM ↑	SAM ↓	CC ↑	RMSE ↓	ERGAS ↓	Note
Full model (LGD-Net)	44.896	0.984	3.288	0.992	0.007	1.685	Full model
w/o Bilinear interaction	44.201	0.983	3.375	0.992	0.008	1.754	Replaced by single linear interaction
w/o Graph Propagation	43.972	0.982	3.423	0.992	0.008	1.774	Removed inter-scale discrepancy-aware propagation
w/o Edge gating	44.385	0.983	3.353	0.992	0.007	1.740	Retained residual propagation but removed gating
w/o Leader refinement	43.818	0.982	3.462	0.992	0.009	1.894	Removed leader-guided feedback to multi-scale nodes (-1.08 dB)
w/o Continuous dynamics	32.725	0.908	4.854	0.958	0.024	4.437	Replaced continuous ODE with standard residual stacking (-12.17 dB)

Key Findings

• Continuous dynamics form the cornerstone of performance: Replacing the continuous dynamical module with discrete residual stacking causes a catastrophic collapse in reconstruction performance—PSNR drops by 12.17 dB (from 44.896 to 32.725 dB) and ERGAS surges from 1.685 to 4.437. This highlights the vital role of smooth, continuous ODE trajectory integration in stabilizing cross-modal interactions.
• Sensitivity to integration steps and step size: With \(T=1\), leader evolution is insufficient (PSNR is only 36.142 dB). Optimal performance is attained at \(T=2\) (41.856 dB), whereas expanding to \(T=3\) or \(T=4\) induces redundant feature mixing and slight degradation (41.242 dB and 40.562 dB). The scaling factor \(s=1.2\) demonstrates a similar convex profile.
• Favorable accuracy-efficiency trade-off: LGD-Net requires only 2.39M parameters, 25.73 GFLOPs, and 34.18 ms per-image latency. Compared to heavy models such as FPFNet (47.34M parameters, 321.47 GFLOPs, 75.19 ms latency), LGD-Net slashes parameters by 95% while achieving substantial gains across all benchmarks.

Highlights & Insights

• Explicit decoupling of global interaction and local reconstruction: Summarizing global context into compact leaders allows continuous-depth ODE integration without the need to deepen the entire feature backbone.
• Bilinear coupling with continuous RK2 dynamics: Integrating additive and multiplicative cross-modal terms under learnable Runge-Kutta numerical integration provides principled, mathematically stable feature trajectory evolution.
• Discrepancy-aware inter-scale graph gating: Regulating residual feature propagation via local adjacent-scale difference ensures that smooth homogeneous regions avoid excessive distortion while edge boundaries receive sharp spatial cues.

Limitations & Future Work

• Static hyper-parameter configuration for ODE solver: Solver steps \(T\) and step scale \(s\) are presently fixed empirically. Dynamically adapting the numerical integration trajectory based on local scene complexity presents a promising future extension.
• Sensitivity to spatial co-registration offsets: The current framework relies on the simulated Wald protocol with strictly aligned LRHS and PAN inputs. In real-world satellite missions, slight spatial misalignment and parallax errors may challenge the leader-guided modulation, warranting unaligned or self-aligning extensions.

Related Work & Insights

• vs HyperPNN / FPFNet (CNN baselines): Standard CNNs accumulate cross-modal interactions implicitly through successive receptive field stacking, leading to spectral distortion in deep layers; LGD-Net explicitly preserves spectral integrity through compact leader dynamics.
• vs HyperTransformer / Lformer (Transformer baselines): Transformers rely on full spatial self-attention maps with high quadratic complexity; LGD-Net restricts continuous cross-modal dynamics to compact leader vectors, cutting memory and computational overhead.
• vs Diffusion-based Fusion Models: Diffusion architectures require dozens of stochastic sampling steps during inference; LGD-Net completes continuous integration in just 2 RK2 solver steps within tens of milliseconds, making it practical for real-time remote sensing pipelines.

Rating

• Novelty: ⭐⭐⭐⭐☆ Introduces leader-guided continuous dynamical state evolution to hyperspectral pansharpening, elegantly decoupling global cross-modal interaction from local depth.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across three representative benchmarks, complete metric coverage, and thorough ablation of ODE integration steps.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous mathematical formulation, lucid diagrams, and clear motivation.
• Value: ⭐⭐⭐⭐☆ Offers an efficient, scalable paradigm for multi-modal remote sensing restoration and image super-resolution.

Related Papers

• [ECCV 2026] Context-Aware Joint Alignment for Cross-Scene Hyperspectral Image Classification
• [CVPR 2026] Geo2: Geometry-Guided Cross-view Geo-Localization and Image Synthesis
• [ECCV 2026] Compact Low-Cost Hyperspectral Imaging via Angular-to-Spectral Diversity Conversion
• [CVPR 2026] Cross-modal Fuzzy Alignment Network for Text-Aerial Person Retrieval and A Large-scale Benchmark
• [ECCV 2026] CRD-Net: Frequency-Adaptive Feature Injection and Change Decoupling for Building Damage Assessment