MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training

Conference: ECCV 2026
Paper: ECCV official page
Project: MMEarth-Bench
Area: Remote Sensing
Keywords: Earth observation, multimodal reconstruction, test-time training, geographic generalization, long-tail distributions

TL;DR

MMEarth-Bench evaluates geographic generalization with five global environmental tasks and twelve aligned modalities, then adapts encoders by reconstructing these modalities at test time; relative to joint training, TTT-MMR improves aggregate results across tasks and both test splits, with geographic batching particularly helpful for long-tail samples.

Background & Motivation

Many remote sensing models are pretrained on large unlabeled Earth observation datasets, but exposure to global imagery does not guarantee that environmental prediction rules learned in one region transfer to another. Labels for biomass and soil properties often depend on sparse, geographically uneven field measurements, while similar optical appearances may correspond to different ecological conditions. Performance on random splits alone cannot adequately expose geographic failures or establish whether additional sensors, climate variables, and terrain information resolve prediction ambiguities.

Existing benchmarks typically provide few modalities per task or cover limited regions, making it difficult to compare multimodal models under consistent global conditions. This paper therefore constructs datasets with twelve shared modalities and holds out all of Africa as a geographic test domain. The experiments also highlight another inefficiency: even when downstream data includes many modalities, a pretrained encoder can accept only its supported inputs. Discarding the remaining modalities leaves available environmental information unused during deployment.

Redesigning every encoder to ingest all modalities would sacrifice model compatibility, while pseudo-labels derived from target predictions risk reinforcing existing errors. Instead, the paper uses observed auxiliary modalities as supervision, such as canopy-height reconstruction error when adapting biomass representations, without requiring the true biomass of test samples. Core Idea: preserve each encoder's compatible inputs, turn all available modalities into test-time reconstruction targets, and combine per-modality gradient normalization with geographic batching to obtain targeted, more stable adaptation signals.

Method

Overall Architecture

The paper contributes both an evaluation benchmark and an adaptation method, rather than a new remote sensing foundation model. MMEarth-Bench defines the tasks, modalities, and geographic splits. TTT-MMR attaches a main-task decoder and a modality decoder to an existing encoder, jointly trains them, updates the encoder on test batches without main-task labels, and then predicts environmental variables or species occurrence.

The encoder always receives its supported modality subset, whereas the twelve auxiliary modalities serve as reconstruction targets. In the diagram, training labels are used only during joint training; main-task labels are unavailable during test-time adaptation. TTT-MMR-Geo changes batch composition, not the encoder's input branches.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    benchmark["Global multimodal<br/>benchmark"] -->|Non-Africa training data and labels| joint["Dual-head joint<br/>training"]
    benchmark -->|Test observations and coordinates| batch["Geographic proximity<br/>batching"]
    joint -->|Initial model for each batch| adapt["Balanced reconstruction<br/>adaptation"]
    batch -->|Batch without main-task labels| adapt
    adapt -->|Fixed main-task head| output["Environmental task<br/>predictions"]
    output -->|Reset to jointly trained state for next batch| adapt

Key Designs

1. Global multimodal benchmark: evaluate label scarcity, modality ambiguity, and geographic shifts together

Each tile contains \(128\times128\) pixels at 10 m ground resolution, covering approximately 1.6 square kilometers. Tiles do not overlap within a task, and the data is single-timestamp rather than a time series. The five tasks are pixel-level aboveground biomass regression; tile-level soil nitrogen, soil organic carbon, and soil pH regression; and multi-label occurrence prediction for 100 terrestrial mammal species. Table 2 reports 18,393, 5,643, 7,982, 8,508, and 36,410 tiles, respectively. Biomass comes from GEDI, soil properties from WoSIS records at 0–5 cm depth, and species labels from IUCN range polygons. The species task therefore does not directly verify field sightings at a particular time.

The twelve modalities comprise six pixel-level sources: Sentinel-2, Sentinel-1, ASTER GDEM, ETH Canopy Height, Dynamic World, and ESA WorldCover; and six tile-level sources: precipitation, temperature, geolocation, Sentinel-2 date, biome, and ecoregion. Here, modalities include metadata and derived products, not twelve independent sensors. All African tiles form the geographic test set; the remaining tiles are split 70%/15%/15% into training, validation, and random testing, with nested 5% and 50% training subsets. This evaluates transfer after fine-tuning globally pretrained models with non-African labels; it does not guarantee that pretraining excluded unlabeled African data.

2. Dual-head joint training: connect environmental prediction and modality reconstruction through shared representations

One encoder feeds both a main-task head and a linear modality decoder. For tile-level tasks, the main-task head uses global average pooling, LayerNorm, and a fully connected layer. For pixel-level biomass, it bilinearly upsamples the embeddings and applies a \(1\times1\) convolution. The modality decoder likewise upsamples embeddings and uses a \(1\times1\) convolution to reconstruct all modality channels, followed by global average pooling for tile-level modalities. Continuous modalities use MSE and categorical modalities use cross-entropy. This provides a common auxiliary interface across tasks and backbones without requiring every modality to enter the encoder.

During joint training, the encoder receives the sum of the main-task loss and the mean modality reconstruction loss, while each head learns its corresponding objective. The purpose is not to pretrain another universal fusion network, but to make reconstruction errors act through a learned decoder on representations relevant to the main task. Keeping this training stage but performing no test-time parameter updates defines the JT baseline. Comparing JT against TTT therefore isolates deployment-time adaptation rather than attributing all auxiliary-training benefits to TTT.

3. Geographic proximity batching: make each update serve a similar local environment

Standard TTT-MMR randomly partitions test tiles into non-overlapping batches. TTT-MMR-Geo instead uses recursive spatial partitioning, similar to a k-d tree, to organize nearby tiles into geographically contiguous batches, with batch size 8 in the experiments. Random batches provide stronger regularization because an update must accommodate diverse regions, but gradients from rare ecological conditions can be diluted by common samples. Geographic batches are more likely to group similar tiles and specialize the update to a local environment.

This batching neither selects examples using test labels nor uses coordinates to predict pseudo-labels. Spatial proximity is only a proxy for sample similarity; the model and reconstruction objective remain unchanged. Its benefit depends on the task and backbone: local specialization can help the long tail, but also removes some constraints supplied by geographically diverse samples. Geographic batching should therefore not be assumed to outperform random batching everywhere.

4. Balanced reconstruction adaptation: prevent a large-scale modality from dominating encoder updates

At test time, the method first computes a batch-averaged reconstruction loss for each modality and differentiates it with respect to encoder parameters. Each modality's gradient is normalized separately before averaging across available modalities and applying an SGD update. Missing modalities are excluded from the adaptation signal. Temperature, terrain, and categorical distributions have different units and loss scales, so directly summing losses can let the largest gradients dominate. The method balances gradient contributions; it does not establish that all modalities are equally reliable for the main task.

Let \(R_m\) denote the batch-averaged reconstruction loss for modality \(m\), \(\theta\) the encoder parameters, and \(\mathcal{M}_{\mathrm{avail}}\) the available modalities. The update mechanism can be summarized as:

\[ {}\theta \leftarrow \theta - \frac{\lambda}{|\mathcal{M}_{\mathrm{avail}}|} \sum_{m\in\mathcal{M}_{\mathrm{avail}}} \frac{\nabla_\theta R_m}{\|\nabla_\theta R_m\|}. \]

This restates the normalize-then-average mechanism from the text. Equation formatting in the cached extraction is damaged, so no particular norm type or zero-gradient stabilization term is inferred. The modality decoder is frozen during adaptation, forcing reconstruction errors to change the encoder's representation; the main-task head is not updated either. After predicting the current batch, the encoder is reset to its post-joint-training state, preventing adaptation from one batch from contaminating the next. This is not continual online learning across the entire test set.

A Worked Example

Consider predicting biomass in an African region with a backbone that accepts only RGB. During training, it learns biomass prediction from non-African labels while reconstructing canopy height, terrain, climate, and other modalities from its RGB representation. Auxiliary modalities do not turn its input layer into a twelve-modality encoder.

At deployment, TTT-MMR-Geo groups up to 8 nearby tiles. The model reconstructs modalities from RGB, compares them with available observations such as canopy height, updates the encoder using the rule above, and predicts through the fixed biomass head. True biomass never enters this process, and the encoder is reset before the next region. This illustrates the data flow, not a claim of an unreported error reduction for a particular tile.

Loss & Training

Joint training uses MSE for the four regression tasks and BCE with logits for species occurrence, alongside the mean modality reconstruction loss. Training updates the encoder and both decoders; testing updates only the encoder using reconstruction signals. Gradient normalization should not be confused with applying a common normalization to raw modality values.

Test-time adaptation uses SGD, learning rate \(10^{-2}\), and batch size 8. On validation batches, the method tries up to 5 iterations, then uses the mean of the batches' best iteration counts during testing. Test labels do not determine stopping, and this does not mean every test batch necessarily runs 5 updates. The training checkpoint is also selected by validation performance.

The benchmark compares 8 pretrained models with 2 randomly initialized ConvNeXtV2A variants. The all-modality variant one-hot encodes categorical modalities and broadcasts tile-level variables over spatial dimensions, producing a 926-channel input. This is a strong supervised baseline, distinct from TTT-MMR's use of modalities as reconstruction targets. All backbones use linear task heads and full-parameter fine-tuning. Inputs follow the corresponding pretrained model's normalization or training-set statistics. Experiments use an NVIDIA H200 140GB GPU.

Key Experimental Results

Main Results

Regression is evaluated with \(R^2\) and species occurrence with mAP. The following random-test results are selected from Table 5. Values are ranks among the three training/adaptation strategies, averaged over the five tasks and random seeds; lower is better, and uncertainty is standard error. These are neither overall backbone rankings nor performance percentages.

Backbone	JT	TTT-MMR	TTT-MMR-Geo
ConvNeXtV2A	2.9 ± 0.1	2.1 ± 0.1	1.1 ± 0.1
Scale-MAE	3.0 ± 0.0	1.8 ± 0.1	1.2 ± 0.1
DINOv3 Sat	3.0 ± 0.0	1.7 ± 0.1	1.3 ± 0.1
MPMAE	3.0 ± 0.0	1.6 ± 0.1	1.4 ± 0.1
TerraMind	3.0 ± 0.0	1.5 ± 0.1	1.5 ± 0.1
Galileo	2.2 ± 0.2	1.7 ± 0.1	2.1 ± 0.2

Additional modalities need not become encoder inputs to help prediction: RGB backbones can benefit as well. However, Galileo's random-test results show that geographic batching improving over JT does not imply superiority over random batching.

Ablation Study

Table 5 also compares batching strategies on the geographic test split. This is an adaptation-strategy analysis, not an invented numerical ablation removing gradient normalization or an individual modality.

Backbone	JT	Random batching: TTT-MMR	Geographic batching: TTT-MMR-Geo
ConvNeXtV2A	3.0 ± 0.0	1.5 ± 0.1	1.5 ± 0.1
Scale-MAE	2.8 ± 0.1	1.9 ± 0.2	1.3 ± 0.1
DINOv3 Sat	3.0 ± 0.0	1.5 ± 0.1	1.5 ± 0.1
MPMAE	2.7 ± 0.2	2.0 ± 0.1	1.3 ± 0.2
TerraMind	2.9 ± 0.1	1.7 ± 0.1	1.3 ± 0.2
Galileo	2.9 ± 0.1	1.6 ± 0.1	1.5 ± 0.2

Key Findings

• Figure 7 applies one-sided Wilcoxon tests to model/seed distributions for each task and split. All reach \(p<0.05\) after Holm-Bonferroni correction. This supports aggregate improvements, not strictly positive gains for every individual run.
• With full training data and random testing, the randomly initialized all-modality ConvNeXtV2A-MM ranks 1st by average rank across tasks; with only 5% of training data, it still ranks 4th. Additional observations can matter more than extensive pretraining, although pretraining has a clearer advantage with fewer labels.
• Four of the five tasks, excluding biomass, exhibit substantial geographic degradation. With 5% training data, no model achieves positive \(R^2\) on the random-test soil organic carbon task. TTT does not fully resolve geographic generalization or low-shot learning.
• Figure 8 analyzes the long tail across all models with one seed: geographic batching reduces underestimation for rare regression targets and helps rarer species more on the random split. Precise residual values are not estimated from the damaged figure-text extraction.

Highlights & Insights

• Treating extra modalities as supervision rather than inputs lets incompatible backbones exploit deployment-time information. Information is transferred through adaptation gradients without redesigning the multimodal fusion architecture.
• Geographic batching makes batch composition part of the adaptation algorithm. It uses real spatial structure to preserve update directions for rare samples instead of merely changing loss weights.
• Both JT and the randomly initialized all-modality baseline are important controls. The former separates auxiliary training from test-time updates, while the latter prevents benefits from richer inputs from being attributed entirely to pretraining.

Limitations & Future Work

• The authors position the datasets as a systematic benchmark, not a route to the best task-specific model. Single-timestamp evaluation does not represent temporal remote sensing, and geographic transfer uses Africa as the sole continent-wide held-out domain.
• The method requires auxiliary observations and back-propagation at test time. Missing modalities can be excluded, but the main text does not sufficiently quantify adaptation under extensive missingness, misregistration, or poor-quality auxiliary products.
• Equalized gradients reduce scale dominance but do not guarantee alignment with the main-task objective. The paper attributes greater stability to normalization, but the available main text lacks an independent numerical ablation quantifying its contribution.
• The cache contains the main paper and references, not the cited appendix. Complete hyperparameters, linear-probing results, and additional model-level gains are outside this note's evidence scope. The main text also does not support precise deployment-latency or back-propagation-cost comparisons.
• Licenses differ: Table 2 lists CC BY for biomass, CC BY-NC for the soil tasks, and separate terms of use for species. Applications must check licenses and label semantics individually; species ranges should not be treated as field-verified occurrence ground truth.

Related Work & Insights

• vs MMEarth / MPMAE: The paper retains the twelve-modality concept but extends reconstruction from pretraining to test-time adaptation across backbones. MPMAE benefits despite already being pretrained on these modalities, suggesting that adaptation is not simply exposure to previously unseen modalities.
• vs TENT: TENT minimizes prediction entropy through batch-normalization-related updates and is naturally suited to classification. This method reconstructs observed modalities and updates the encoder, covering four regression tasks and Transformers without BatchNorm.
• vs UDA-SS: UDA-SS incorporates unlabeled target-domain data during training. Here, downstream joint training does not access target-domain data, which is used only at test time. Whether global pretraining previously encountered that region is a separate issue.
• vs READ: READ addresses reliability bias while encoding and fusing modalities; this method does not require auxiliary modalities to be encoded as inputs. A useful extension would preserve this compatibility while jointly accounting for modality reliability and geographic similarity during adaptation.

Rating

• Novelty: 4/5. A unified global benchmark and model-agnostic multimodal test supervision are useful contributions built on established joint-training and TTT frameworks.
• Experimental Thoroughness: 4/5. Five tasks, two splits, ten model configurations, and statistical tests provide breadth, but the main text lacks separate quantitative studies of normalization, missing modalities, and runtime.
• Writing Quality: 4/5. The problem, benchmark, and method connect clearly, although average ranks, absolute performance, and aggregate improvements require careful distinction.
• Value: 4/5. The benchmark and adaptation baselines better reflect geographic deployment conditions, while practical applicability depends on test-time modality availability.