SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation

Conference: ECCV 2026
Paper: ECCV 2026 Oral/Poster
Code: https://gitlab.citius.gal/hpc4rs/simpler
Area: Multimodal VLM
Keywords: Representation Similarity, Structured Pruning, Earth Observation, Parameter-Efficient Fine-Tuning, Pre-Fine-Tuning Compression

TL;DR

Addressing the heavy computational and deployment overhead of Earth Observation foundation models, SIMPLER introduces pre-fine-tuning layer pruning guided by CKA representation similarity on unlabeled downstream data, pruning up to 79% of parameters while preserving 94%~97% baseline accuracy with 2.1× training and 2.6× inference speedups.

Background & Motivation

Fine-tuning and deploying large-scale Earth Observation (EO) foundation models (such as the 300M-parameter ViT Prithvi-EO-2) incur immense computational burdens. Adapting these models for downstream tasks—like crop type mapping, marine debris tracking, or oil spill detection—demands hundreds of GPU hours; for instance, fine-tuning Prithvi-EO-2 on BigEarthNetv2 requires roughly 2.81 hours on a 4-GPU H200 cluster. At inference time, latency and memory footprints scale linearly with transformer depth, presenting a severe deployment bottleneck for critical edge platforms like orbital satellites, drones, and battery-powered monitoring sensors.

Existing efficiency techniques address either training cost or inference latency in isolation. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA and adapters, constrain updates to low-rank subspaces, significantly reducing training memory and gradient overhead; however, they keep all transformer layers active during inference, providing zero deployment acceleration. Conversely, traditional structured pruning compresses models for fast inference but operates strictly post-hoc: engineers must first pay the full fine-tuning cost, inspect adapted weight statistics, and iteratively prune and retrain. Furthermore, magnitude-based pruning heuristics frequently misjudge layer importance by discarding small-weight layers that execute indispensable semantic transformations. No existing paradigm simultaneously slashes both fine-tuning cost and deployment latency within a unified framework.

This paper tackles the challenge from a distinct vantage point: pre-trained vision transformers naturally exhibit representational stabilization in deeper layers when processing domain-specific downstream imagery. Layers deep in the network produce redundant, highly correlated representations long before any task-specific adaptation begins. The core idea is to compute layer-wise Centered Kernel Alignment (CKA) similarity matrices on small batches of unlabeled downstream imagery prior to fine-tuning, leveraging an automated variation-gap scoring rule to identify the optimal depth cutoff \(c^*\) without hyperparameters or gradients, thereby enabling direct, efficient fine-tuning and high-throughput deployment of a compact sub-network.

Method

Overall Architecture

SIMPLER rethinks foundation model adaptation by shifting structural pruning from post-adaptation weight space to pre-fine-tuning representation space. The end-to-end framework operates in two distinct stages: Pre-analysis and Downstream Fine-tuning. In the pre-analysis stage, a small batch of unlabeled downstream target images (500 samples are sufficient) is passed through the frozen pre-trained backbone in a single forward pass. Layer-wise representations are extracted, and a Centered Kernel Alignment (CKA) similarity matrix is constructed. A hyperparameter-free scoring function evaluates representational variability across retained versus candidate pruned layers to automatically pinpoint the optimal depth cutoff \(c^*\). In the subsequent fine-tuning stage, the truncated dense backbone retaining only the first \(c^*\) layers is coupled with a downstream task head and adapted via standard full fine-tuning or combined with low-rank adaptation (LoRA).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Small unlabeled task sample S<br/>(500 images, CPU / single forward pass)"] --> B["1. Layer Representation Extraction<br/>Extract frozen ViT hidden states {h_1, ..., h_L}"]
    B --> C["2. Similarity Matrix Computation<br/>Compute pairwise layer CKA matrix Z (orthogonal invariance)"]
    C --> D["3. Automated Cutoff Scoring<br/>Maximize diversity-stability gap score(c)=Δ_TL - Δ_BR"]
    D --> E["4. Compact Backbone Truncation<br/>Prune redundant deep blocks, retain first c* layers F_c*"]
    E --> F["5. Downstream Adaptation<br/>Full fine-tuning or compound LoRA tuning on task head H_D"]

Key Designs

1. Representation Similarity Profiling: Exploiting Intrinsic Deep Stabilization Rather than relying on empirical weight magnitudes that can discard critical low-magnitude transformations, or compute-heavy gradient sensitivity metrics that require backpropagation through the full network, SIMPLER uses representation similarity to directly measure operational redundancy across layers. Given a pre-trained model \(\mathcal{F}_{\text{pre}}\) with \(L\) layers and an unlabeled sample set \(S \subset \mathcal{D}\), hidden state representations \(h_\ell\) are extracted for each layer \(\ell \in \{1, \dots, L\}\). SIMPLER analyzes three similarity metrics: Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), and nearest-neighbor Jaccard similarity. CKA demonstrates superior performance due to its invariance to orthogonal transformations and smooth gradient properties across layer depth. When adjacent deep layers exhibit high CKA similarity on downstream distributions, the gradient descent loss landscape flattens along these aligned trajectories, rendering the redundant deep layers ineffective during fine-tuning while wasting memory and computation.

2. Automated Layer Selection Criterion: Parameter-Free Depth Optimization To eliminate arbitrary manual thresholds and costly hyperparameter sweeps, SIMPLER introduces a matrix block variation metric. Given the \(L \times L\) similarity matrix \(\mathbf{Z}\), any candidate layer cutoff \(c \in \{2, \dots, L-2\}\) partitions \(\mathbf{Z}\) into a top-left block \(\mathbf{Z}_{TL} = \mathbf{Z}[0:c-1, 0:c-1] \in \mathbb{R}^{c \times c}\) (retained early/intermediate layers) and a bottom-right block \(\mathbf{Z}_{BR} = \mathbf{Z}[c:L, c:L] \in \mathbb{R}^{(L-c) \times (L-c)}\) (pruned deep layers). For any \(k \times k\) block \(\mathbf{M}\), the mean absolute consecutive-row difference is defined as:

\[\delta(\mathbf{M}) = \frac{1}{(k-1)k} \sum_{i=0}^{k-2} \sum_{j=0}^{k-1} |\mathbf{M}_{i,j} - \mathbf{M}_{i+1,j}|\]

The variability measures are calculated as:

\[\Delta_{TL} = \delta(\mathbf{Z}_{TL}), \qquad \Delta_{BR} = \delta(\mathbf{Z}_{BR})\]

Here, \(\Delta_{TL}\) captures representational diversity among retained layers (higher diversity indicates progressive hierarchical feature extraction), while \(\Delta_{BR}\) quantifies representational stability among pruned layers (lower values denote high similarity and redundant computations). The optimal architecture depth \(c^*\) is determined automatically by maximizing the gap:

\[c^* = \arg\max_{c \in \{2, \dots, L-2\}} (\Delta_{TL} - \Delta_{BR})\]

This formulation operates strictly on the geometry of the unlabelled representation space and requires zero validation labels or manual tuning.

3. Orthogonal Adaptation Strategies: Compounding Depth Reduction with PEFT After isolating the compact sub-network \(\mathcal{F}_{c^*}\) containing the first \(c^*\) layers, SIMPLER yields a standard dense architecture rather than an irregular sparse network, eliminating any dependency on specialized sparse matrix runtimes. Because SIMPLER operates in architectural depth space rather than parameter update space, it is strictly orthogonal to parameter-efficient fine-tuning paradigms. Practitioners can either execute Full Fine-tuning on \(\mathcal{F}_{c^*}\) and task head \(\mathcal{H}_D\) to achieve maximum throughput, or deploy SIMPLER with LoRA. Combining architectural truncation with low-rank weight updates reduces trainable parameters down to 0.1%~0.2% of the original model, dramatically reducing GPU VRAM requirements during adaptation while simultaneously speeding up inference by up to 2.9×.

A Worked Example

Consider adapting the 24-layer, 300M-parameter Prithvi-EO-2 foundation model to the MADOS marine pollution segmentation task: 1. Sampling & Profiling: 500 unlabeled Sentinel-2 multispectral patches are fed through frozen Prithvi-EO-2. Hidden states are gathered and evaluated via CKA in CPU RAM (taking 218 seconds), generating the \(24 \times 24\) similarity matrix \(\mathbf{Z}\). 2. Matrix Partitioning: Inspecting \(\mathbf{Z}\), layers 0~5 show substantial divergence (low similarity values), layers 6~15 act as an intermediate transition, and layers 16~24 collapse into an intense block of near-identical features (\(CKA \approx 1.0\)). Evaluating \(\text{score}(c) = \Delta_{TL} - \Delta_{BR}\) across all candidate depths reveals an unambiguous global maximum at \(c^* = 5\). 3. Pruning & Adaptation: The model is sliced to retain only blocks 1~5, discarding blocks 6~24. Total model parameters fall from 303.90M to 64.57M (a 79% reduction). When fine-tuned on MADOS, training wall-clock time drops from 15.90 min to 7.46 min (2.1× faster), inference throughput increases from 33.02 img/s to 88.72 img/s (2.6× faster), while maintaining a 62.8% mIoU compared to the full 66.9% baseline.

Loss & Training

SIMPLER introduces no additional regularization terms or auxiliary loss functions during pre-analysis. In the downstream adaptation phase, standard task-specific losses are employed: - Semantic segmentation (MADOS): Cross-entropy loss or weak supervision multi-label segmentation loss; - Multi-label land cover classification (BigEarthNetv2): Multi-label binary cross-entropy loss over 19 co-occurring classes; - Time series crop classification (Sen4Map): Multi-class temporal cross-entropy. All downstream training hyperparameters (AdamW optimizer, learning rate schedules, weight decay) directly replicate standard baseline configurations without task-specific re-tuning.

Key Experimental Results

Main Results

The authors benchmark SIMPLER across three distinct Earth Observation downstream tasks with Prithvi-EO-2 (300M & 600M), alongside cross-architecture evaluations on TerraMind (multimodal EO) and ViT-MAE (RGB natural imagery).

Task / Dataset	Model / Method	Total Params (M)	Trainable Params (M)	Train Time (min)	Inf Throughput (img/s)	Primary Metric	Performance Retention
MADOS (Segmentation)	300M Full Baseline	303.90	303.90	15.90±4.80	33.02±1.94	mIoU: 66.9±2.5%	100%
	300M Baseline + LoRA	306.32	2.42	11.77±2.17	31.62±1.13	mIoU: 59.6±1.5%	89.1%
	300M Post-hoc Prune 20%	240.92	240.92	24.34±5.09	35.33±4.02	mIoU: 58.4±1.6%	87.3%
	300M Post-hoc Prune 40%	177.94	177.94	22.51±4.83	47.03±6.03	mIoU: 47.9±3.7%	71.6%
	300M SIMPLER (Ours)	64.57	64.57	7.46±1.62	88.72±15.04	mIoU: 62.8±1.2%	93.9%
	300M SIMPLER + LoRA (Ours)	65.12	0.55	4.31±0.28	79.51±14.44	mIoU: 60.4±1.4%	90.3%
	600M Full Baseline	631.21	631.21	29.20±8.11	16.28±0.31	mIoU: 69.6±2.2%	100%
	600M SIMPLER (Ours)	80.24	80.24	7.70±1.90	77.26±11.27	mIoU: 62.2±2.0%	89.4%
BigEarthNetv2 (Classification)	300M Baseline	303.91	303.91	168.87±6.60	37.67±0.04	mAP: 73.4±0.5%	100%
	300M SIMPLER (Ours)	51.98	51.98	40.66±1.64	110.05±0.36	mAP: 71.2±0.3%	97.0%
	300M SIMPLER + LoRA (Ours)	52.43	0.45	92.29±16.33	105.11±0.42	mAP: 70.1±0.2%	95.5%
Sen4Map (Time Series)	300M Baseline	303.90	303.90	133.81±8.48	4.72±0.00	Macro-F1: 66.6±0.3%	100%
	300M SIMPLER (Ours)	89.76	89.76	55.62±3.83	15.55±0.03	Macro-F1: 63.8±0.4%	95.8%

Ablation Study

Table 5 in the paper evaluates the impact of representation similarity metrics and examines how pre-training contributes to the performance of pruned sub-networks on MADOS (Prithvi-EO-2 300M):

Ablation Group	Configuration	Selected Cutoff	Params (M)	mIoU (%)	Pixel Acc (%)	Note
Similarity Metrics	Baseline (Full 24 layers)	24 blocks	303.90	66.9±2.5	95.3±1.2	Full model upper bound
	SIMPLER (Jaccard / SVCCA)	2 blocks	26.78	50.7±3.4	84.6±0.8	Over-aggressive cutoff, drops 16.2% mIoU
	SIMPLER (CKA, Default)	5 blocks	64.57	62.8±1.2	94.2±1.1	Optimal cutoff, retains 93.9% mIoU
Pre-training Contribution	Baseline (From Scratch)	24 blocks	303.90	46.7±2.4	83.0±0.8	Untrained 24-layer baseline
	Baseline (Fine-tuning)	24 blocks	303.90	66.9±2.5	95.3±1.2	Pre-training brings +43.2% relative gain
	SIMPLER (From Scratch)	5 blocks	64.57	44.1±1.9	81.9±1.3	From-scratch capacity matches full baseline
	SIMPLER (Fine-tuning)	5 blocks	64.57	62.8±1.2	94.2±1.1	Pre-training brings +42.4% relative gain

Key Findings

1. CKA Metric Superiority: Nearest-neighbor Jaccard and SVCCA both aggressively select a cutoff at block 2 (retaining only 26.78M parameters), triggering severe performance degradation (50.7% mIoU). In contrast, CKA reliably pinpoints block 5 (64.57M parameters), achieving 62.8% mIoU (+12.1 points over SVCCA). CKA's invariance to orthogonal transformations and continuous depth-gradient behavior make it the most dependable measure of functional redundancy.
2. Effective Preservation of Pre-trained Knowledge: When trained from scratch without pre-training, the 5-block sub-network performs almost identically to the full 24-block model (44.1% vs 46.7% mIoU), proving that deeper layers provide negligible structural learning capacity on downstream tasks. Upon fine-tuning with pre-trained weights, both the 5-block and 24-block architectures enjoy an equivalent ~42%–43% relative performance jump, confirming that SIMPLER successfully preserves the most valuable pre-trained representations.
3. The "Reduce Large" Superiority: On the multimodal TerraMind model (Table 6), pruning TerraMind-Large down to 53.23M parameters yields an mIoU of 58.8%, substantially outperforming the native unpruned TerraMind-Small baseline (22.37M parameters, 53.0% mIoU) by 5.8 points. This demonstrates that adapting a pruned large foundation model captures substantially richer representations than independently training and fine-tuning natively smaller foundation models.

Highlights & Insights

• Pre-Fine-Tuning Architecture Selection Paradigm: Breaks the traditional paradigm of post-hoc compression (Fine-tune \(\rightarrow\) Prune \(\rightarrow\) Retrain) by diagnosing architectural redundancy before adaptation starts, avoiding sunk training costs.
• Hyperparameter-Free Matrix Variation Formulation: Leverages the difference between intra-block variation metrics (\(\Delta_{TL} - \Delta_{BR}\)) to automatically detect the elbow point where layer evolution gives way to representational plateauing, eliminating heuristic tuning.
• Seamless Orthogonal Fusion with PEFT: Combining layer pruning with LoRA achieves compound efficiency: reducing depth slashes latency and FLOPs for inference, while low-rank matrices slash trainable parameters down to 0.45M for memory-constrained edge adaptation.

Limitations & Future Work

• Reliance on Masked Autoencoder (MAE) Pre-training Dynamics: The progressive layer stabilization exploited by SIMPLER naturally emerges under MAE-style reconstruction objectives. Models pre-trained with contrastive learning (e.g., SoftCon) or explicit collapse prevention (e.g., DINOv3 with KoLeo regularization) often exhibit oscillating or non-stabilizing similarity heatmaps, where block-diagonal redundancy is less pronounced.
• Fixed Sequential Truncation Scope: The current formulation focuses on pruning consecutive tail layers (\(c^*\) to \(L\)). Extending the similarity metric to support non-contiguous layer skipping or multi-branch architectures remains an open direction.
• Performance Trade-offs in Long-Range Temporal Reasoning: On Sen4Map time-series crop classification, full-depth LoRA baseline achieves 66.9% Macro-F1 compared to SIMPLER's 63.8%, indicating that tasks heavily reliant on complex multi-temporal attention dependencies are more sensitive to model depth truncation than spatial visual tasks.

Related Work & Insights

• vs. PEFT (LoRA / AdaptFormer): PEFT methods operate purely in parameter space to accelerate training, but retain full depth during inference; SIMPLER operates in architectural depth space, unlocking substantial inference speedups, and natively compounds with LoRA.
• vs. Post-hoc Structured Pruning (DepGraph / L1-Pruning): Traditional pruning incurs full fine-tuning costs before pruning and depends on weight magnitude heuristics that often discard critical low-weight layers; SIMPLER performs pre-adaptation layer selection via representation geometry.
• vs. Dynamic Early Exiting (BranchyNet / LGViT): Dynamic exit methods require auxiliary routing classifiers and introduce execution branching overhead on batch-oriented hardware; SIMPLER outputs standard dense feedforward networks that execute seamlessly on generic PyTorch and edge runtimes (e.g., NVIDIA Jetson).

Rating

• Novelty: ⭐⭐⭐⭐☆ (Pioneering use of representation similarity as an a priori pruning criterion before fine-tuning)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across segmentation, multi-label, and time series tasks, spanning Prithvi-EO-2, TerraMind, ViT-MAE, and edge Jetson hardware)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clean mathematical formulation, cohesive narrative, and well-designed comparative diagrams)
• Value: ⭐⭐⭐⭐⭐ (Provides an immediate, practical roadmap for deploying foundation models on satellites and edge devices)