MorphJEPA: Morphology-Aware Latent Prediction for Hyperspectral Images

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/amartya-ray/MorphJEPA
Area: Remote Sensing
Keywords: Hyperspectral Imaging, Self-Supervised Learning, Joint-Embedding Predictive Architecture (JEPA), Mathematical Morphology, Learning from Limited Supervision (LLS)

TL;DR

Addressing the vulnerability of pixel reconstruction to sensor noise and the fragility of contrastive views in hyperspectral imaging, MorphJEPA introduces the first JEPA tailored to HSI by predicting embeddings of pixel-aligned morphological opening/closing targets, coupled with a disentangled cross-attention encoder and a context-anchored distribution regularizer (SIGReg), yielding substantial gains in 1/5/10-shot adaptation and cross-scene transfer.

Background & Motivation

Hyperspectral imaging (HSI) acquires contiguous narrow spectral bands across spatial dimensions, capturing detailed physical reflectance signatures that allow fine-grained discrimination of surface materials for precision agriculture, environmental monitoring, and urban planning. However, expert-level pixel annotation is exceptionally labor-intensive and costly, creating severe label scarcity in operational domains. This motivates the paradigm of learning from limited supervision (LLS): learning transferable representations from abundant unlabeled HSI cubes in the target domain, and subsequently adapting the representation with only a handful of labeled patches per class (e.g., 1, 5, or 10 samples). Nevertheless, the extreme dimensionality of HSI triggers the Hughes phenomenon (overfitting under limited supervision), while dense patch sampling introduces massive spatial-spectral redundancy that often conceals subtle material cues.

Prevailing HSI self-supervised learning (SSL) paradigms predominantly rely on contrastive or reconstruction-based frameworks, both of which suffer from fundamental flaws when applied to hyperspectral data. Contrastive learning depends heavily on handcrafted data augmentations; however, because the continuous spectral curve itself serves as the physical identity of a material, generic operations like band masking, random dropping, or noise perturbation risk distorting critical absorption signatures and violating semantic consistency. On the other hand, masked autoencoding (MAE) methods minimize pixel-level reconstruction errors, which forces the network to allocate extensive representation capacity to memorizing dataset-specific high-frequency sensor noise and atmospheric perturbations that fail to transfer across sensors, seasons, or geographic regions. While Joint-Embedding Predictive Architectures (JEPAs) circumvent pixel reconstruction by predicting representations in latent space, generic spatial or band masking disrupts essential spatial-spectral coherence, producing ill-posed prediction targets.

The central insight of this work is that in HSI scene analysis, macroscopic morphology—spatial connectivity, boundary contours, and entity geometry—exhibits significantly higher invariance across atmospheric and sensor variations than high-frequency textures. Mathematical morphological opening and closing operators provide a principled, idempotent mechanism to isolate these structural invariants by suppressing localized spatial fluctuations while strictly preserving underlying geometric skeletons and spectral alignments. The core idea is to repurpose mathematical morphology from a conventional input-side preprocessing filter into a self-supervised latent prediction pretext within a JEPA framework, enforcing a directional structural bottleneck where predicting morphologically simplified targets from raw context compels the encoder to suppress unpredictable sensor noise and focus on transferable topological invariants.

Method

Overall Architecture

MorphJEPA learns robust, transferable spatial-spectral representations via latent cross-view prediction across structurally simplified, pixel-aligned views. Given an unlabeled hyperspectral patch \(x \in \mathbb{R}^{S \times S \times B}\) (dimension-reduced via PCA to 15 components), the model constructs three pixel-aligned target views: a geometric view \(x^{(g)}\) derived via random rotations and flips, and two morphology-filtered views \(x^{(o)}\) and \(x^{(c)}\) generated via per-band morphological opening and closing operators. A student encoder \(f_\theta\) maps the raw context patch \(x\) to a context embedding \(z_c\). Simultaneously, a momentum-updated Exponential Moving Average (EMA) teacher encoder \(f_{\theta'}\) extracts target latent representations (\(z_{geo}, z_{open}, z_{close}\)) with stop-gradient enforcement.

Both student and teacher share a lightweight disentangled spatial-spectral encoder architecture, wherein multi-scale spatial features and pure 1D spectral mixing features are conditionally integrated via a cross-attention bottleneck. The student context embedding \(z_c\) is mapped through three dedicated MLP predictors (\(h_{geo}, h_{open}, h_{close}\)) to match the respective teacher target embeddings under an \(L_2\) predictive distance. To prevent dimensional collapse driven by high spatial-spectral patch correlations, an asymmetric Sketched Isotropic Gaussian Regularizer (SIGReg) is applied exclusively to the student context embeddings, anchoring the latent representation space to a well-conditioned, isotropic Gaussian distribution. Downstream classification is performed by freezing the pretrained encoder and training a single linear head on few-shot labeled target data.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Raw HSI Patch x<br/>S×S×B (after PCA)"] --> VIEW["Morphological & Geometric View Generation<br/>Per-band opening/closing + geometric transforms"]

    subgraph STU["Student Encoding Branch (Gradients Active)"]
        direction TB
        IN --> ENC_S["Disentangled Spatial-Spectral Encoder f_θ<br/>Dilated spatial convolutions + 1×1 spectral conv"]
        ENC_S --> CA_S["Cross-Attention Bottleneck<br/>Spatial Query attends to Spectral Key/Value"]
        CA_S --> ZC["Context Embedding z_c"]
    end

    subgraph TEA["EMA Teacher Branch (Stop-Gradient)"]
        direction TB
        VIEW --> ENC_T["EMA Teacher Encoder f_θ'<br/>Momentum coefficient m=0.996"]
        ENC_T --> T_EMB["Target Latent Embeddings<br/>z_geo, z_open, z_close"]
    end

    subgraph HEADS["Directional Prediction & Manifold Regularization"]
        direction TB
        ZC --> PRED["Task-Specific Predictors (h_geo, h_open, h_close)<br/>Predict target embeddings in latent space"]
        PRED --> LOSS_PRED["Latent Prediction Loss L_pred<br/>Sum of squared L2 distances"]
        T_EMB --> LOSS_PRED
        ZC --> SIGREG["Context-Anchored Regularizer L_reg<br/>SIGReg matching to isotropic Gaussian N(0, I)"]
    end

    LOSS_PRED --> TOTAL["Joint Pretraining Objective<br/>L_total = L_pred + λ L_reg"]
    SIGREG --> TOTAL

Key Designs

1. Morphological Latent Prediction Targets: Directional Structural Bottleneck via Geometric Topology Generic masking destroys the continuous spatial-spectral fabric of hyperspectral data, while raw pixel reconstruction overfits high-frequency noise. MorphJEPA replaces destructive masking with per-band mathematical morphological filtering using a flat \(3 \times 3\) structuring element. Specifically, opening (erosion followed by dilation) eliminates bright spatial spikes and isolates connected components, while closing (dilation followed by erosion) bridges dark narrow channels and seals micro-voids. Predicting the latent representations of these idempotent, smoothed targets from the raw context patch \(x\) establishes a directional structural bottleneck: because unpredictable high-frequency sensor noise is eliminated from the teacher targets, the student cannot and need not predict it, thereby compelling the encoder to isolate macro-geometry, physical boundaries, and stable topological invariants that generalize across scenes.

2. Disentangled Spatial-Spectral Encoder: Spatial Queries Attending to Spectral Semantics Monolithic 3D-CNNs and vision transformers tightly entangle spatial and spectral processing, incurring severe overfitting in low-shot adaptation and causing spatial morphological transformations to implicitly corrupt spectral material signatures. MorphJEPA addresses this with a compact 0.18M-parameter factorized encoder. The spatial branch captures multi-scale geometry by concatenating standard \(3 \times 3\) convolutions with dilated \(3 \times 3\) convolutions (\(d=2\)), producing a spatial feature map \(F_{spa} \in \mathbb{R}^{S \times S \times D}\). Concurrently, the spectral branch executes point-wise channel mixing via a \(1 \times 1\) convolution without spatial aggregation, preserving pure material signatures in \(F_{spec} \in \mathbb{R}^{S \times S \times D}\). Both maps are flattened into \(T = S^2\) tokens and fused via a cross-attention bottleneck, where spatial tokens serve as queries \(Q\) to modulate spectral keys \(K\) and values \(V\): $\(H = Q + \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{D_h}}\right)V\)$ This ensures that morphological variations remain confined to the spatial pathway, while pure physical absorption spectra anchor material identity.

3. Context-Anchored Latent Manifold Regularization: Preventing Dimensional Collapse Dense overlapping spatial-spectral patch extraction induces severe latent feature correlations. Without explicit manifold constraints, predictive representations easily collapse into a low-rank, anisotropic subspace. While LeJEPA proposed an isotropic Gaussian regularizer (SIGReg) based on empirical characteristic function matching, applying it symmetrically across all views without a momentum teacher causes catastrophic manifold degeneration under severe HSI correlation (collapsing into a degenerate 1D curve with only 24.27% OA on Houston 2013). MorphJEPA resolves this by pairing an EMA teacher with an asymmetric, context-anchored SIGReg applied strictly to student context embeddings \(Z_c = \{z_c^{(k)}\}_{k=1}^M\). By matching the empirical context distribution against an isotropic prior \(\mathcal{N}(0, I)\), it constrains the covariance trace \(\mathrm{Tr}(\Sigma)\) and prevents ill-conditioning. Theoretically, this tightens the Rademacher generalization bound for downstream \(N\)-shot linear probing \(\mathcal{O}\big(B\sqrt{\mathrm{Tr}(\Sigma)/N}\big)\), ensuring well-conditioned feature separation under extreme label scarcity.

Loss & Training

The pretraining objective combines multi-view latent prediction with context distribution regularization. Given the predicted latent targets \(\hat{z}_v = h_v(z_c)\) and teacher representations \(z_v\) for view \(v \in \{geo, open, close\}\), the prediction loss is defined as: $\(\mathcal{L}_{pred} = \|\hat{z}_{geo} - z_{geo}\|_2^2 + \|\hat{z}_{open} - z_{open}\|_2^2 + \|\hat{z}_{close} - z_{close}\|_2^2\)$ The geometric target enforces spatial equivariance and absolute material consistency, preventing semantic drift, while the opening/closing targets supervise topological structure. The overall pretraining objective is: $\(\mathcal{L}_{total} = \mathcal{L}_{pred} + \lambda \mathcal{L}_{reg}\)$ where \(\lambda = 0.05\). Optimization uses AdamW with an initial learning rate of \(2 \times 10^{-4}\), weight decay of \(10^{-4}\), cosine annealing to \(10^{-6}\) over 100 epochs, and an EMA momentum of 0.996. Downstream classification evaluates linear probing: the pretrained encoder is completely frozen, and a linear classifier is trained on \(N\)-shot labeled target samples (\(N \in \{1, 5, 10\}\)) for 100 epochs using AdamW (learning rate \(10^{-3}\), weight decay 0.01).

Key Experimental Results

Main Results

MorphJEPA is evaluated on two intra-scene benchmarks (Houston 2013, Trento) and two cross-scene generalization benchmarks (Houston 2013 \(\to\) 2018 cross-sensor temporal transfer; HyRank Dioni \(\to\) Loukia cross-geographic spatial transfer). Evaluation metrics include Overall Accuracy (OA, %), Average Accuracy (AA, %), and Kappa coefficient (\(\kappa \times 100\)), averaged over 10 independent runs.

Benchmark & Protocol	Metric	MorphJEPA (Ours)	Second Best	Gain (vs. Second Best)	Standard SSL Baseline (MAE / I-JEPA)
Houston 2013 (5-shot intra-scene)	OA (%) AA (%) \(\kappa\)	74.73 77.53 72.61	71.85 (SC-EADNet) 75.51 (SC-EADNet) 69.28 (SC-EADNet)	+2.88% OA +2.02% AA +3.33 \(\kappa\)	MAE: 68.45% / 71.23% I-JEPA: 48.57% / 51.22% LeJEPA: 24.27% (collapsed)
Houston 2013 (1-shot intra-scene)	OA (%) AA (%)	52.06 59.65	50.73 (SC-EADNet) 55.72 (SC-EADNet)	+1.33% OA +3.93% AA	MAE: 41.24% / 44.47% I-JEPA: 29.83% / 32.38%
Trento (5-shot intra-scene)	OA (%) AA (%) \(\kappa\)	89.62 84.40 86.20	87.26 (DMVL) 82.89 (CM-SCON) 84.28 (CM-SCON)	+2.36% OA +1.51% AA +1.92 \(\kappa\)	MAE: 83.17% / 78.44% I-JEPA: 68.43% / 64.18%
Houston 2013 \(\to\) 2018 (5-shot cross-sensor adaptation)	OA (%) AA (%) \(\kappa\)	67.21 62.43 51.33	66.01 (GGCL) 61.10 (SC-EADNet) 47.28 (SC-EADNet)	+1.20% OA +1.33% AA +4.05 \(\kappa\)	MAE: 37.53% / 37.22% SpectralDINO: 50.03%
HyRank Dioni \(\to\) Loukia (5-shot cross-scene adaptation)	OA (%) AA (%) \(\kappa\)	60.37 71.22 54.11	61.43 (SC-EADNet) 68.01 (SpectralDINO) 52.04 (SC-EADNet)	-1.06% OA +3.21% AA +2.07 \(\kappa\)	MAE: 43.80% / 44.50% GGCL: 50.79% / 62.44%

In computational efficiency, MorphJEPA requires only 0.18M parameters, 0.04 GFLOPs, and achieves an inference latency of 0.98 ms per patch, dramatically outperforming heavy models like DMVL (24.62M / 2.12 GFLOPs / 5.87 ms) and SpectralDINO (21.55M / 0.75 GFLOPs / 12.16 ms).

Ablation Study

The impact of pretraining loss components and architectural fusion mechanisms was analyzed under the 5-shot regime on Houston 2013 (intra-scene) and HyRank Dioni \(\to\) Loukia (cross-scene).

Config ID	Loss Components (\(\mathcal{L}_{geo}, \mathcal{L}_{open}, \mathcal{L}_{close}\))	Context Reg. (\(\mathcal{L}_{reg}\))	Backbone / Fusion Strategy	Houston 2013 OA (%)	HyRank Adaptation OA (%)	Key Takeaways & Mechanism Analysis
(i)	Geo only (\(\mathcal{L}_{geo}\))	\(\times\)	Disentangled Cross-Attn	53.23 ± 2.40	36.52 ± 4.12	Pure geometric equivariance lacks explicit topological structural priors
(ii)	Morphology only (\(\mathcal{L}_{open}+\mathcal{L}_{close}\))	\(\times\)	Disentangled Cross-Attn	68.97 ± 3.72	54.85 ± 5.10	Morphology yields huge gains (+15.74% OA) but higher variance due to global semantic drift
(iii)	Geo + Morphology	\(\times\)	Disentangled Cross-Attn	73.97 ± 1.62	59.76 ± 3.90	Geometric target acts as a semantic anchor, reducing standard deviation from 3.72 to 1.62
(iv)	Full Model (All \(\mathcal{L}_{pred}\))	\(\checkmark\) (SIGReg)	Disentangled Cross-Attn	74.73 ± 1.11	60.37 ± 3.15	Context SIGReg achieves optimal isotropic conditioning, reducing variance by 31%
(A)	Supervised from scratch	—	Disentangled from scratch	31.92 ± 13.20	22.15 ± 14.50	Extreme low-shot data causes catastrophic overfitting and high optimization variance
(B)	Full MorphJEPA SSL	\(\checkmark\)	Monolithic 3D-CNN (4.10M)	64.12 ± 1.85	48.50 ± 2.90	Entangled spatial-spectral processing overfits limited downstream labels
(C)	Full MorphJEPA SSL	\(\checkmark\)	SpectralFormer (0.38M)	69.50 ± 1.40	52.40 ± 2.55	Generic vision transformer lacks local inductive bias suited for HSI cubes
(D)	Full MorphJEPA SSL	\(\checkmark\)	Concat Fusion (0.12M)	72.10 ± 1.15	56.90 ± 2.20	Direct concatenation lacks cross-modal modulation between space and spectrum
(E)	Full MorphJEPA SSL	\(\checkmark\)	CBAM Attention (0.15M)	73.05 ± 1.19	58.05 ± 2.35	Simple channel/spatial re-weighting fails to capture fine-grained topological boundaries

Key Findings

• Complementarity of Morphological Operators: Applying opening or closing alone yields suboptimal performance compared to their combination. Atomic erosion or dilation alters entity areas and boundary coordinates, introducing systematic predictive bias. In contrast, idempotent compound opening/closing operators capture a comprehensive topological profile.
• Symmetric vs. Asymmetric Manifold Regularization: LeJEPA suffers complete dimensional collapse on HSI data due to symmetric regularization without momentum anchoring (t-SNE collapses into a continuous 1D thread, achieving only 24.27% OA). MorphJEPA's asymmetric setup—anchoring student context embeddings via SIGReg while relying on an EMA teacher—preserves class discriminability and yields a well-conditioned manifold (\(\kappa = 1.14\)).
• Trade-off between Regional Cohesion and Fine Linear Features: Qualitative thematic maps on Trento demonstrate that MorphJEPA completely eliminates the "salt-and-pepper" classification artifacts seen in competing methods, producing cohesive regional structures (e.g., Apple Trees, Vineyard). The trade-off is a slight smoothing of ultra-narrow linear structures (such as single-pixel roads), which is a deliberate regularization driving superior few-shot transfer.

Highlights & Insights

• Elevating Morphology to a Self-Supervised Latent Pretext: Instead of treating mathematical morphology as a feature engineering trick or early-layer filter, MorphJEPA pioneers its role as a latent prediction target in JEPA, creating a directional structural bottleneck that naturally rejects high-frequency sensor noise.
• Asymmetric SIGReg Regularization for High-Dimensional Remote Sensing: Uncovers why symmetric non-contrastive regularizers collapse under spatially and spectrally correlated patch regimes, presenting an asymmetric student-only Gaussian anchoring strategy that theoretically bounds few-shot linear complexity.
• High Accuracy with Micro-Scale Parameters: Achieves new state-of-the-art transfer performance across challenging temporal and geographic shifts with only 0.18M parameters and 0.98 ms latency, offering high utility for real-time edge processing on airborne or satellite platforms.

Limitations & Future Work

• Over-Smoothing of Highly Directional Linear Structures: The use of isotropic \(3 \times 3\) structuring elements can smooth out elongated, continuous thin features such as narrow roads or irrigation canals. Future investigations could explore path-based or directional morphological operators (Path Opening).
• Absence of Sub-Pixel Texture Reconstruction: By operating purely in latent space to enforce macroscopic semantic invariance, MorphJEPA discards microscopic pixel-level residuals, which may limit effectiveness in fine-grained sub-pixel spectral unmixing tasks. A hybrid predictive-reconstructive objective could offer a balanced solution.

Related Work & Insights

• vs. I-JEPA (Assran et al., CVPR 2023): I-JEPA masks large contiguous spatial blocks in RGB images and predicts them from context; in HSI, spatial masking severs spectral continuity and critical material signatures. MorphJEPA maintains full pixel alignment while applying morphological filtering to define structural targets.
• vs. LeJEPA (Balestriero & LeCun, arXiv 2025): LeJEPA discards momentum teachers and applies symmetric SIGReg across all embeddings, which collapses into a 1D degenerate subspace on correlated HSI cubes. MorphJEPA retains an EMA teacher and restricts SIGReg to student context embeddings to preserve well-conditioned target manifolds.
• vs. SC-EADNet & SpectralDINO (SOTA HSI SSL & Few-Shot): SC-EADNet relies on fragile multi-scale contrastive views that degrade under temporal sensor shifts; SpectralDINO requires high parameter overhead (21.55M). MorphJEPA achieves superior cross-scene transfer with a compact 0.18M factorized architecture.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Pioneering integration of mathematical morphology with JEPA latent prediction; highly original and domain-appropriate)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Thorough 1/5/10-shot evaluations across intra-scene, temporal cross-sensor, and geographic cross-scene shifts with detailed ablations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Crystal-clear motivation, robust theoretical backing via optimal transport and Rademacher bounds, and insightful visualizations)
• Value: ⭐⭐⭐⭐⭐ (Directly addresses high annotation costs in HSI with an ultra-lightweight, edge-deployable framework)