Skip to content

OneHSI: A Unified Hyperspectral Foundation Model with Physical Consistency

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Remote Sensing
Keywords: Hyperspectral Imaging, Foundation Model, Cross-Sensor Generalization, Physical Consistency, Scale Equivariance

TL;DR

Addressing device-dependent coupling in hyperspectral learning caused by band discretization discrepancy, multi-order-of-magnitude spatial resolution shifts, and severe cross-domain data imbalance, OneHSI establishes a unified, physically consistent foundation model through continuous wavelength coordinate reparameterization, scale-equivariant MoE transformation, and cross-domain adversarial representation equalization.

Background & Motivation

Hyperspectral imaging (HSI) offers material-level physical perception far beyond conventional RGB vision by sampling contiguous narrow spectral bands across hundreds of wavelengths. It plays an indispensable role in satellite earth observation, precision agriculture, autonomous driving perception, and intraoperative cancer assessment. However, deep learning for HSI has long been constrained by a device-dependent paradigm: sensing configurations vary radically across instruments, spanning hundreds of meters in satellite remote sensing to sub-micrometer scales in histopathology, with incompatible band counts and spectral response functions (SRFs). Consequently, models trained on a specific sensor fail dramatically when deployed to another hardware configuration or discretization setting, severely limiting model reproducibility and cross-device generalizability.

Although recent vision foundation models have been extended to hyperspectral imageryβ€”such as SpectralGPT, HyperSIGMA, SpectralEarth, and HyperFree via masked image modeling (MAE) self-supervised pretrainingβ€”they remain predominantly trained on relatively homogeneous satellite datasets and implicitly assume fixed spectral bands and spatial scales. When deployed to ground-level scenarios like medical imaging or autonomous driving where band distributions and spatial resolutions differ substantially, their performance deteriorates sharply. Fundamentally, while raw sensor values fluctuate due to illumination geometry, atmospheric absorption, and sensor noise, the underlying material reflectance curves (absorption features, spectral peaks, and band ratios) are governed by objective chemical and physical properties rather than the sensing device itself. Conventional deep learning architectures discard this intrinsic continuity by treating spectral bands and spatial grids as arbitrary discrete tensor coordinates.

This breakdown of cross-sensor consistency stems directly from three core bottlenecks: spectral misalignment across sensor-specific wavelength coordinates (C1), spatial-scale feature shifts induced by orders-of-magnitude resolution changes (C2), and severe domain imbalance where satellite remote sensing data heavily dominates ground-level data (C3). The angle of attack in this paper is to build inductive biases directly aligned with material physics, treating discrete bands and resolutions as non-uniform samplings of an underlying continuous physical domain. Core idea: align hyperspectral representation learning with underlying material reflectance physics through a unified framework integrating Spectral Coordinate Reparameterization (SCORE), Scale-Equivariant Adaptive Transformation (SEAT), and Cross-Domain Representation Equalization (CORE) to learn truly sensor-invariant representations.

Method

Overall Architecture

OneHSI processes a hyperspectral cube \(X_{m,i} \in \mathbb{R}^{H \times W \times C_m}\) from domain \(m\), together with its sensor wavelength coordinates \(\lambda_m \in \mathbb{R}^{C_m}\) and physical spatial resolution \(s_m\) (meters/pixel). The architecture addresses cross-sensor heterogeneity systematically across data, spectral, and spatial levels: CORE first samples balanced mini-batches with equal domain contributions; SCORE projects discrete spectral channels into a unified continuous wavelength feature space and applies lightweight statistical modulation; following the MAE paradigm, the masked tokens enter a ViT backbone where self-attention is shared globally to preserve scale-invariant relational structures while feed-forward layers are modulated by physical scale-routed Mixture-of-Experts (SEAT); finally, a lightweight decoder reconstructs the cube in the physically aligned feature space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous HSI Input<br/>X, wavelength Ξ», spatial scale s"] --> B["Cross-Domain Representation Equalization<br/>Balanced 4-domain sampling + adversarial GRL"]
    B --> C["Spectral Coordinate Reparameterization<br/>Continuous wavelength mapping + FiLM modulation"]
    C --> D["Patch & Mask Tokenizer<br/>Visible physical patch tokens"]
    D --> E["Scale-Equivariant Adaptive Transformation<br/>Shared self-attention + scale-guided MoE experts"]
    E --> F["Lightweight Reconstruction Decoder<br/>Reconstruct in aligned physical space"]
    F --> G["Joint Multi-Objective Supervision<br/>Reconstruction + Expert balance + Domain adversarial"]

Key Designs

1. Spectral Coordinate Reparameterization: Projecting discrete channels into continuous wavelength space

Traditional models treat spectral bands as discrete tensor channel dimensions, causing projection matrices to collapse whenever band counts or channel orderings change. SCORE treats each spectral channel as a point sampled from an underlying continuous function over wavelength. Given sensor wavelength coordinates \(\lambda_m \in \mathbb{R}^{C_m}\), wavelengths are first normalized against the global bounds \([\lambda_{\min}, \lambda_{\max}]\) computed across all pretraining domains:

\[\hat{\lambda}_m = \frac{\lambda_m - \lambda_{\min}}{\lambda_{\max} - \lambda_{\min}}\]

A shared continuous mapping network \(\Phi_{\text{map}}: \mathbb{R} \to \mathbb{R}^D\) is evaluated independently on each normalized wavelength \(\hat{\lambda}_m^{(c)}\) to dynamically construct a sensor-specific projection basis \(W_m \in \mathbb{R}^{C_m \times D}\). Multiplying the input cube by \(W_m\) produces a physically aligned tensor \(X'_{m,i} = X_{m,i} W_m \in \mathbb{R}^{H \times W \times D}\) that completely eliminates discrete channel dependencies. To compensate for sensor-level radiometric and statistical biases, SCORE further applies a lightweight FiLM modulation: spatial adaptive average pooling on \(X'_{m,i}\) extracts a global context vector \(c_{m,i} \in \mathbb{R}^D\), which is mapped via an affine transformation to scale \(\gamma_{m,i}\) and shift \(\delta_{m,i}\) parameters, producing the modulated feature \(\tilde{X}_{m,i} = X'_{m,i} \odot (1 + \gamma_{m,i}) + \delta_{m,i}\).

2. Scale-Equivariant Adaptive Transformation: Decoupling relational attention from scale-conditioned filtering

Varying physical scale across orders of magnitude (from tens of meters per pixel in satellites to sub-micrometers in microscopy) radically changes spatial frequency and local textures, but preserves long-range semantic and topological relations among scene regions. SEAT incorporates this insight into Transformer blocks: multi-head self-attention is shared globally across all domains to capture scale-invariant relational dependencies, whereas scale-specific adaptations are strictly confined to the feed-forward network (FFN) path through physical scale-guided MoE experts.

To handle multiplicative resolution jumps smoothly, SEAT operates in logarithmic scale space \(s'_m = \log_{10}(s_m)\). It instantiates \(E\) scale-aware experts, each defined by a learnable scale center \(\mu_e\) and bandwidth \(\sigma_e^2\). The gating affinity between domain \(m\) and expert \(e\) is parameterized by a smooth Gaussian kernel:

\[g_{m,e} = -\frac{(s'_m - \mu_e)^2}{\sigma_e^2}\]

Applying softmax normalization yields gating probability \(G_{m,e}\). The top-\(K_{\text{exp}}\) (default \(K_{\text{exp}}=2\)) experts are selected and their weights are renormalized. This formulation enables smooth interpolation across continuous resolutions while providing dedicated specialized filtering for extreme resolution regimes.

3. Cross-Domain Representation Equalization: Countering data imbalance and sensor domain bias

Public hyperspectral data is heavily biased toward satellite remote sensing, which accounts for the vast majority of available samples, while driving and medical HSI data remain scarce. Standard joint training allows dominant remote sensing gradients to overwhelm the shared representations, leading to representation collapse on smaller domains. CORE enforces mini-batch level domain equalization: pretraining data is partitioned into four major pools (remote, proximal, medical, driving), and each mini-batch uniformly draws \(K_{\text{dom}}\) samples from each pool, guaranteeing an equal \(1/M\) gradient weighting. Furthermore, a domain classifier equipped with a Gradient Reversal Layer (GRL, \(\mathcal{R}_\eta\)) is attached to the pooled token representations, penalizing the encoder for retaining sensor-identifying artifacts and encouraging purely physics-based domain-invariant representations.

Loss & Training

Pretraining is supervised end-to-end by a composite objective with three regularizers targeting distinct heterogeneity dimensions:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{rec}} + \alpha \cdot \mathcal{L}_{\text{bal}} + \beta \cdot \mathcal{L}_{\text{dal}}\]
  • Physically Aligned Reconstruction Loss \(\mathcal{L}_{\text{rec}}\): Computes mean squared error (MSE) between decoded predictions \(\hat{X}_{m,i,p}\) and ground-truth features \(\tilde{X}_{m,i,p}\) over masked patch indices \(\mathcal{M}_{m,i}\). The target \(\tilde{X}_{m,i,p}\) is detached from the backward graph to avoid trivial representation collapse:
\[\mathcal{L}_{\text{rec}} = \frac{1}{|\mathcal{B}|} \sum_{(m,i) \in \mathcal{B}} \frac{1}{|\mathcal{M}_{m,i}|} \sum_{p \in \mathcal{M}_{m,i}} \|\tilde{X}_{m,i,p} - \hat{X}_{m,i,p}\|_2^2\]
  • Expert Balancing Loss \(\mathcal{L}_{\text{bal}}\): Prevents gating collapse by regularizing the dot product between mean routing probabilities \(P_e^l\) and domain assignment fractions \(f_e^l\) across all \(L\) layers, encouraging learned scale anchors \(\mu_e\) to distribute uniformly along the log-scale axis (\(\alpha = 0.10\)):
\[\mathcal{L}_{\text{bal}} = \frac{E}{L} \sum_{l=1}^L \sum_{e=1}^E f_e^l \cdot P_e^l\]
  • Domain-Adversarial Loss \(\mathcal{L}_{\text{dal}}\): Employs cross-entropy on global token representation \(z_{m,i}\) through gradient reversal layer \(\mathcal{R}_\eta\) against domain labels \(y_m\) (\(\beta = 0.05\)). The model uses a ViT-Huge backbone, pretrained on 8 NVIDIA A800 GPUs for 100 epochs (100,000 iterations) with AdamW, cosine annealing, and a 0.75 masking ratio.

Key Experimental Results

Main Results

OneHSI is evaluated across 16 downstream benchmarks covering 4 sensing domains and 6 distinct hyperspectral tasks: classification (HC), segmentation (HS), object detection (HOD), change detection (HCD), anomaly detection (HAD), and salient object detection (HSD). Representative cross-domain and multi-scale results include:

Dataset Domain / Task Metric OneHSI (Ours) Prev. SOTA Gain
OHID-1 Remote / Urban Land Cover (HS) OA (%) / KA (%) 92.50 / 89.83 91.93 / 87.52 (MambaHSI) +0.57 / +2.31
NaSC-TG2 Remote / Natural Scene Classif. (HC) OA (%) / AA (%) 95.53 / 95.03 94.66 / 94.01 (HyperFree) +0.87 / +1.02
M2SODAI Remote / Maritime Object Detect. (HOD) Precision (%) / [email protected] (%) 81.02 / 78.24 76.58 / 74.03 (HyperFree) +4.44 / +4.21
HSI-BAY Remote / Multitemporal Change (HCD) OA (%) / F1 (%) 98.37 / 98.07 97.91 / 97.68 (CSDBF) +0.46 / +0.39
HRSSD Remote / Salient Object Detect. (HSD) \(F_\beta\) (%) / \(S_\alpha\) (%) 58.64 / 68.41 55.62 / 67.56 (DSSN) +3.02 / +0.85
DeepHS Proximal / Fruit Ripeness Classif. (HC) OA (%) / KA (%) 72.36 / 54.06 68.73 / 52.13 (MambaHSI) +3.63 / +1.93
HZDR Proximal / PCB Defect Segment. (HS) OA (%) / mIoU (%) 77.02 / 62.15 75.23 / 59.42 (MambaHSI) +1.79 / +2.73
LIB-HSI Proximal / Building Facades (HS) OA (%) / mIoU (%) 93.08 / 86.72 92.14 / 85.06 (MambaHSI) +0.94 / +1.66
HSI-AgriFoodAnomaly Proximal / Food Anomaly Detect. (HAD) OA (%) / F1 (%) 95.41 / 91.87 94.27 / 85.71 (GT-HAD) +1.14 / +6.16
HSIDermoscopy Medical / Melanoma Detection (HC) OA (%) / KA (%) 76.28 / 61.38 73.34 / 49.72 (PWS-CS) +2.94 / +11.66
HistologyHSI-GB Medical / Glioblastoma Classif. (HC) OA (%) / KA (%) 88.36 / 76.85 86.43 / 72.01 (MambaHSI) +1.93 / +4.84
HeiPorSPECTRAL Medical / Organ Segmentation (HS) OA (%) / KA (%) 96.81 / 95.54 95.42 / 93.94 (MambaHSI) +1.39 / +1.60
Hyper-Drive Driving / Unstructured Scene (HS) OA (%) / KA (%) 86.91 / 77.48 84.26 / 75.85 (MambaHSI) +2.65 / +1.63
Hyperspectral City V2 Driving / Urban Road Segment. (HS) OA (%) / AA (%) 88.38 / 53.48 87.53 / 52.37 (MambaHSI) +0.85 / +1.11
HyperRoad Driving / Complex Road Segment. (HS) OA (%) / AA (%) 98.54 / 97.96 98.02 / 97.43 (MambaHSI) +0.52 / +0.53

Under controlled head-to-head comparison against foundation models re-trained with identical data and protocols: - Efficiency: OneHSI has 953.8M parameters with only 363.25G FLOPs, significantly more efficient than HyperSIGMA (1715.7M / 841.81G FLOPs) and HyperFree (744.5M / 510.37G FLOPs). - Frozen Feature Quality (Frozen OA %): On OHID-1, OneHSI scores 90.65% (vs. 86.47% for HyperSIGMA and 86.14% for HyperFree); on HeiPorSPECTRAL, OneHSI achieves 94.10% (vs. 91.63% and 90.46%), proving superior representation quality out of the box.

Ablation Study

Table 1: Ablation on Spectral Alignment (SA) and Statistical Modulation (SM) in SCORE (Metric: OA %)

Variant Spectral Alignment (SA) Statistical Modulation (SM) OHID-1 (Remote) LIB-HSI (Proximal) HeiPor (Medical) HyperRoad (Driving)
OneHSI (Full Model) βœ“ βœ“ 92.50 93.08 96.81 98.54
w/o SM (Only continuous alignment) βœ“ – 91.65 90.12 95.41 95.83
w/ Proj. (Replaced with CNN projection) – βœ“ 90.27 89.13 91.94 93.72
w/ Inter. (Replaced with linear interpolation) – βœ“ 85.10 86.65 91.35 91.19

Table 2: Ablation on SEAT Routing and Auxiliary Loss Objectives (Metric: OA %)

Variant Architectural / Loss Configuration OHID-1 (Remote) LIB-HSI (Proximal) HeiPor (Medical) HyperRoad (Driving)
OneHSI (Full Model) Scale-aware MoE (Top-2) + \(\mathcal{L}_{\text{bal}}\) + \(\mathcal{L}_{\text{dal}}\) 92.50 93.08 96.81 98.54
SEAT, Top-1 Scale-aware MoE (Single expert) 91.95 92.60 96.25 98.20
Base MoE, Top-2 Standard token routing MoE (Top-2) 90.80 91.45 94.55 97.40
Base MoE, Top-1 Standard token routing MoE (Single expert) 90.45 91.12 94.20 97.15
No MoE Dense FFN backbone 89.85 90.50 93.15 96.60
w/o \(\mathcal{L}_{\text{bal}}\) Without expert load balancing loss 90.68 92.01 93.94 97.09
w/o \(\mathcal{L}_{\text{dal}}\) Without domain-adversarial loss 89.73 89.24 92.37 96.14
SEAT-only SEAT architecture without auxiliary losses 87.41 89.82 92.13 95.78
ViT Baseline Standard ViT (No SCORE / SEAT) 89.72 88.95 92.47 96.31

Key Findings

  1. Physical wavelength mapping vastly outperforms linear interpolation: Replacing the continuous coordinate mapping with conventional linear wavelength interpolation leads to catastrophic performance drops (plunging by 7.40% on OHID-1 to 85.10% and by 6.43% on LIB-HSI to 86.65%). Non-linear physical absorption characteristics cannot be synthesized via naive geometric interpolation.
  2. Scale-aware routing gains stem from resolution priors, not capacity: Standard MoE with token-based routing achieves only marginal improvements (+0.60% to +1.40%) over dense FFNs. In contrast, SEAT improves performance by +3.66% on HeiPorSPECTRAL over the dense baseline, proving that physical log-scale guidance is the decisive driver of multi-scale adaptation.
  3. Data efficiency and perturbation robustness: In 1-shot transfer, OneHSI achieves superior performance across all four domains (80.76% on remote, 79.84% on proximal, 85.06% on medical, and 92.37% on driving), exceeding HyperSIGMA by up to 4.98%. Furthermore, under sensor noise (Gaussian, Poisson, and striping), OneHSI drops by only 2.59 OA on average, half the performance degradation of HyperSIGMA (5.03) and HyperFree (5.60).

Highlights & Insights

  • Decoupled relational and scale-adaptive modeling: Retaining globally shared self-attention while delegating scale-specific transformations exclusively to physical scale-routed FFN experts allows the model to scale across orders of magnitude at a remarkably low computational cost (363G FLOPs).
  • Coordinate-free spectral continuous projection: Formulating projection matrix generation as continuous evaluations of \(\Phi_{\text{map}}(\hat{\lambda})\) renders inference latency virtually invariant to the number of input spectral bands, eliminating hardware re-engineering for new sensors.
  • Transferable physical representation paradigm: Grounding multi-sensor representations in continuous physical attributes (wavelength coordinates and physical sampling resolutions) provides a transferable template for other heterogeneous scientific imaging domains such as multispectral radar, astronomical surveys, and acoustic tomography.

Limitations & Future Work

  • Dependency on accurate wavelength metadata: SCORE relies on calibrated central wavelength metadata \(\lambda_m\). Severe metadata corruption, uncalibrated drift, or wide overlapping filter responses could introduce systematic bias into the projection basis.
  • Absence of explicit radiative transfer degradation modeling: The framework assumes material reflectance consistency across scales but does not explicitly embed atmospheric radiative transfer equations (e.g., water vapor absorption bands) or deep tissue light scattering models into the pretraining pipeline.
  • Future Directions: Developing blind wavelength calibration estimators for legacy or uncalibrated imagery, and incorporating physical radiative transfer degradation priors directly into self-supervised pretraining.
  • vs HyperSIGMA & SpectralEarth: Existing foundation models primarily focus on satellite remote sensing (e.g., EnMAP, Sentinel-2), assuming homogeneous sensor settings. OneHSI unifies satellite macro-scales (tens of meters) and biomedical micro-scales (micrometers) in a shared physical continuum.
  • vs HyperFree: While HyperFree utilizes prompt engineering and CNN projections for band adaptation, it lacks physical continuous grounding and incurs heavier computation. OneHSI reduces FLOPs by ~29% while achieving superior frozen representation transfer.
  • vs MambaHSI & S2Mamba: SSM-based approaches excel at modeling long spectral dependencies on fixed sensors, but their discrete state-space parameters cannot transfer across heterogeneous sensor hardware. OneHSI demonstrates that physics-guided attention and scale-aware MoE architectures establish a superior foundation for general cross-hardware HSI modeling.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering work that re-anchors hyperspectral foundation modeling in physical material reflectance consistency via continuous wavelength projection and scale-equivariant MoE.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering 4 major sensing domains, 6 visual tasks, and 16 diverse benchmarks, validated under controlled baselines, few-shot setups, and physical sensor noise.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clean, rigorous mathematical formulations, lucid motivation, and compelling narrative flow connecting physics to deep architecture.
  • Value: ⭐⭐⭐⭐⭐ Resolves decades of sensor-hardware lock-in in hyperspectral imaging, establishing a robust blueprint for universal cross-sensor perception models.