OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/mbzuai-oryx/OpenEarthAgent
Area: Remote Sensing
Keywords: tool-augmented agents, geospatial reasoning, deterministic replay, trajectory supervision, multisource remote sensing

TL;DR

OpenEarthAgent connects visual perception, GIS, spectral indices, and georeferenced raster operations to a unified executor, then fine-tunes Qwen3-4B on replay-validated trajectories, improving strict tool-order matching from 14.71% to 67.24% on its test set while non-generation answer accuracy remains only 45.26%.

Background & Motivation

Remote-sensing questions cannot always be answered by describing an image. Finding the distance between two types of facilities requires defining the geographic boundary, querying points of interest, and computing distances; comparing vegetation or fire-related changes across dates also requires handling acquisition times, spectral indices, and spatial extent. Models such as Prithvi and AnySat learn earth observation representations, while GeoChat and EarthDial enable language-based interaction with remote-sensing imagery. Neither representation quality nor descriptive ability alone guarantees that these analytical steps can be executed correctly.

The difficulty lies particularly in tool dependencies and spatial semantics: a GeoTIFF is a georeferenced raster file, not a place name that can be submitted directly to a geocoder. Pixel counts are not square meters either; conversion requires ground sampling distance. ThinkGeo and Earth-Agent have begun connecting external tools to remote-sensing reasoning, but models still need to select the right tools, supply valid arguments, retain coordinate context, and plan from returned results. A syntactically valid step does not establish that the whole chain meets the task requirements, and a plausible explanation does not establish a physical basis for its geographic operations.

The paper therefore focuses on which trajectories are worth learning, rather than simply increasing the number of questions. It first standardizes heterogeneous data and tool calls, executes candidate trajectories to check geometry, arguments, and spatial consistency, and trains a language controller to reproduce these executable decisions. Core Idea: combine unified geospatial tool interfaces with execution-validated multistep trajectories to turn remote-sensing reasoning from answer generation into learning a traceable, runnable analytical workflow.

Method

Overall Architecture

The input is a natural-language task together with relevant materials, which may include RGB or SAR images, pre/post-event imagery, GIS layers, or georeferenced rasters. Outputs may be numerical or textual answers, annotated images, maps, or derived layers. The model decides the next step, tools perform perception and spatial computation, and their results enter working memory before the model acts again.

The complete approach includes data construction and replay validation during training preparation, together with a tool-feedback loop at inference time. Dashed edges below denote training data or supervision relationships; solid edges denote tool specifications or inference data flow. Trajectory supervision and replay do not mean retraining the model for each test query.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multisource data and annotations"] --> B["Executable Corpus Construction"]
    B -.->|Candidate trajectories| E["Trajectory Supervision and Replay"]
    C["Unified Tool Registry"] --> D["Working Memory and Feedback Execution"]
    C -.->|Tool-controller validation| E
    Q["Query and imagery or layers"] --> D
    D --> T["Tool results and cache"]
    T --> D
    E -.->|Offline supervised training| D
    D --> O["Answer or visualization artifact"]

This is not a newly trained visual encoder that directly consumes every remote-sensing modality. The specifically trained component is the Qwen3-4B-Instruct-2507 language controller; external perception, GIS, and spectral tools jointly provide multimodal capabilities. Controller size therefore does not represent the full system's computational cost.

Key Designs

1. Executable Corpus Construction: establish data availability and grounding before generating trajectories

The RGB/SAR branch first filters out samples lacking the required annotations and harmonizes labels and annotation formats across source datasets. The GIS branch starts from candidate regions and available point-of-interest layers, avoiding questions about objects that are absent. The index branch mines meaningful events from temporal changes in indicators such as NDVI, NBR, and NDBI. NDVI characterizes vegetation, NBR supports burn-related analysis, and NDBI supports built-up-area analysis. These indicators give questions computable surface attributes rather than relying only on visual appearance.

The branches converge into unified JSON records containing image paths, GeoPackage information, spatial attributes, and tool arguments. A language model with task-specific templates and a one-shot exemplar then generates queries and reasoning trajectories. Candidate GIS queries are executed programmatically; generated entries undergo argument, geometry, and spatial-consistency checks, with controlled regeneration and targeted manual verification when necessary. The authors additionally inspect every test sample manually, explicitly mentioning corrections to GSD inconsistencies in area estimates. The final corpus contains 14,538 training samples and 1,169 test samples, with 100,656 training steps and 7,064 test steps. A step denotes an explicit thought-action-observation transition, not a separate question-answer pair.

2. Unified Tool Registry: give heterogeneous geographic operations explicit input-output contracts

Visual detectors return boxes, GIS operators return geometry or distances, and spectral operators return index layers. Free-form prose alone is an unreliable way to connect these results. The registry specifies structured inputs, structured outputs, and an executable function for each tool using consistent JSON contracts. A central orchestrator parses model-generated calls, validates arguments, executes the function, and adds its result to the context. The model learns which interface to invoke next and which existing results to pass, rather than simulating what a tool might return.

There are five tool categories: perceptual tools handle detection, region attributes, counting, segmentation, and change detection; GIS tools handle boundaries, points of interest, distances, and related spatial quantities; spectral tools add index layers, compute temporal changes, and visualize results; georeferenced raster tools extract GeoTIFF bounds and render results on the original raster; utility tools provide calculators, solvers, plotting, text annotation, OCR, retrieval, and termination. The central issue is not tool count but preserving object types, coordinate references, and file references so that one operation's output is a valid input to the next. The authors state that new tools can be registered without retraining, but interface registration does not establish that the model will correctly use every unfamiliar tool.

3. Working Memory and Feedback Execution: preserve completed actions and generated layers in the state

Each decision reads working memory containing the user instruction, previous tool calls, observations, spatial metadata, and execution feedback. After the controller proposes a call, the orchestrator executes it and appends the returned result. The model then continues from the updated state until it invokes Terminate. This changes the task from one-shot long-form generation into an observable interaction. For example, after a layer has been saved successfully, later operations should reference that artifact rather than guess its contents or location again.

Separate from working memory, an execution cache stores derived vector layers, raster subsets, index maps, and geometries for reuse within the same trajectory. The two components have different roles: working memory preserves decision context, whereas the cache preserves actual data artifacts. Caching avoids repeated computation and supports reuse of consistent intermediate states during replay. However, the paper does not establish reproducibility under arbitrary changes to external data sources, nor does it show that caching alone guarantees correct final answers.

4. Trajectory Supervision and Replay: train the action policy without training it to fabricate environment feedback

Before a trajectory enters training, the tool controller executes it again to check argument formatting, coordinate integrity, geometric validity, and full-chain executability. Compared with checking whether JSON parses, replay can expose calls with complete fields that nevertheless reference nonexistent layers or describe spatially invalid operations. Accepted trajectories supply next-action targets for supervised fine-tuning. Historical tool results remain conditioning information, grounding action learning in actual feedback.

The method section describes optimizing only the tool-action policy, with tool observations excluded from the loss. The implementation section describes response-only masking: only assistant-generated tokens receive loss, while prompt text and external tool outputs are masked. Both descriptions separate the policy from its environment, but the main paper does not fully specify the masking granularity of every text field within assistant turns. The method should therefore be understood as supervised learning conditioned on execution feedback, not training the model to memorize tool-generated values as answers, and not a policy discovered through online reinforcement learning.

A Worked Example

The task in Figure 1 is to show museums and malls on a supplied GeoTIFF, find the nearest pair, and annotate the image with their distance. The critical dependency is not recognizing what a museum is, but extracting geographic bounds from the raster before performing GIS operations.

1. After planning, call GetBboxFromGeotiff to extract west, south, east, and north bounds, then pass those bounds to GetAreaBoundary to obtain the GeoPackage used by subsequent GIS operations.
2. Call AddPoisLayer separately with {"tourism": "museum"} and {"shop": "mall"}. The illustrated returns contain 1 point of interest in each category. These are computable layers, not merely objects mentioned in an explanation.
3. Use ComputeDistance to find the closest distance between the layers, overlay results on the original raster with DisplayOnGeotiff, annotate the distance using AddText, and terminate with the output image. The figure presents 9 steps including high-level planning, calls, and completion; these are not 9 distance computations.

The untuned Qwen3-4B instead passes from GeoTIFF directly as an area argument to the geocoder and terminates prematurely when no result is returned. This comparison illustrates learning the correct input dependencies, but one case does not establish general error-recovery ability. The cached figure does not provide a reliably readable final distance, so no distance value is invented here.

Loss & Training

Training follows autoregressive maximum likelihood: conditioned on the task, previous actions, and environment returns, it increases the probability of the validated next action. The cached equations contain obvious extraction damage, so this note retains the mechanism in prose rather than presenting a repaired expression as the authors' exact formula.

The backbone is Qwen3-4B-Instruct-2507, trained with Unsloth's FastLanguageModel on 4 NVIDIA A100 40 GB GPUs for 1 epoch. The learning rate is 2e-5, with a cosine schedule, a 0.05 warmup ratio, batch size 16, and a maximum sequence length of 4096 tokens. Training samples alternate between text and JSON tool interactions; external tool outputs provide context only.

Key Experimental Results

Main Results

The in-house test set contains 1,169 samples, and all compared models serve as interchangeable controllers within the same framework. Step-by-step evaluation checks the next call given an interaction history without actually executing tools, and excludes the initial high-level planning step from validation. End-to-end evaluation executes tools and lets the model continue from their returns. The protocols must not be conflated.

Table 1 selects results from the paper's Table 3; values are percentages. Inst. checks whether a call contains syntactic or logical errors, Tool evaluates tool selection, ArgN checks that required arguments are present, ArgV checks their values, and Summ. evaluates consolidation of historical results into a final response.

Controller	Inst.	Tool	ArgN	ArgV	Summ.
Qwen3-4B-Instruct-2507, untuned	97.34	86.94	84.12	33.55	83.28
GPT-4o	99.18	93.88	85.48	45.80	86.76
o4-mini	83.68	68.33	64.17	37.95	89.48
OpenEarthAgent-4B	99.51	97.18	96.08	62.10	83.64

Table 2 selects end-to-end results from the paper's Table 4; values are percentages. GIS F1 compares selected GIS tools with reference tool sets. AnyOrder requires matching tools and repetition counts but ignores order; SameOrder additionally requires matching sequence order; Unique compares tool sets without repetition counts.

Controller	GIS F1	AnyOrder	SameOrder	Unique	Ans.	Gen.
Qwen3-4B-Instruct-2507, untuned	71.82	16.00	14.71	21.47	13.72	15.86
GPT-4o	95.80	50.81	50.38	55.52	39.22	77.93
GPT-5	91.50	46.96	46.79	47.81	43.88	46.21
OpenEarthAgent-4B	98.52	67.75	67.24	72.71	45.26	75.86

Ans. is answer accuracy for non-generation tasks, judged by gpt-4o-mini with a standardized prompt. Gen. is based on the correctness and execution success of image-generation-related tool calls, not a comprehensive assessment of artifact quality. Consequently, 75.86% is not a measure of remote-sensing image generation quality and is not directly comparable with 45.26% as if they represented the same task type.

Ablation Study

The supplied main paper only states that additional ablations appear in the supplementary material, which is absent from this cache. Table 3 therefore summarizes same-backbone comparisons before and after training, not individual removals of the registry, cache, or replay. It supports the effectiveness of the overall training recipe but cannot isolate any component's causal contribution.

Protocol and metric	Untuned Qwen3-4B	OpenEarthAgent-4B	Gain (percentage points)
In-house test set, step-by-step ArgV	33.55	62.10	28.55
In-house test set, end-to-end SameOrder	14.71	67.24	52.53
In-house test set, end-to-end Ans.	13.72	45.26	31.54
Earth-Agent, step-by-step Tool	60.61	70.33	9.72

The final row comes from the paper's Table 5. The Earth-Agent benchmark contains 248 tasks and 13,729 images; its tools are adapted to the interface and supplied with standardized JSON exemplars. This demonstrates some cross-benchmark transfer, not zero-shot execution in a completely unadapted tool environment. GPT-4o achieves Tool accuracy of 71.65 in the same table, still above the proposed model's 70.33.

Key Findings

• Call formatting was already relatively stable; the large gains occur in argument values and long-chain ordering. This matches the goal of learning tool dependencies rather than only valid JSON.
• SameOrder reaches 67.24%, but Ans. is only 45.26%, showing that following a reference workflow does not ensure a correct answer. Step-by-step Summ. also rises only from 83.28% to 83.64%, without a comparable jump.
• The paper's Table 6 reports average latency of 21.67 seconds on a balanced 60-query subset, versus 42.02 seconds for the backbone and 933.54 seconds for GPT-Agent. These are system latencies under particular tools, tasks, and deployment conditions, not model-only inference speedups.

Highlights & Insights

• Executable trajectories are a more concrete data-quality criterion than detailed explanations. Including coordinates, GSD, and layer references in validation directly constrains spatial prerequisites that textual reasoning can overlook.
• Excluding tool returns from the training target preserves the boundary between model decisions and environmental facts. Other specialist agents can similarly prioritize reliable action supervision instead of memorizing external-system outputs.
• The small controller benefits primarily from alignment to domain workflows. This suggests that a larger controller need not be the first requirement for a specialist tool system, although tool perception quality and execution cost remain part of the system.

Limitations & Future Work

• The main paper has no standalone limitations section, but its results show remaining gaps in argument values, final answers, and image tasks. The conclusion likewise positions the system as a step toward geospatial agents.
• From an experimental-design perspective, the currently verifiable material lacks component ablations separating the effects of data filtering, deterministic replay, caching, and trajectory fine-tuning. Follow-up comparisons should control data volume and computational budget.
• From an evaluation perspective, reference tool order may penalize equivalent solutions, and an LLM judge may introduce bias. Independent numerical checks of distances, areas, coordinate errors, and final layers would strengthen the evidence.
• Replay establishes executability in a given environment, not perpetual correctness of external geographic data. The main paper does not fully develop stress tests for geographic isolation, temporal extrapolation, or changing live data; real disaster and environmental monitoring deployments still require validation.

Related Work & Insights

• vs GeoChat / EarthDial: These works primarily advance language interaction with remote-sensing imagery; OpenEarthAgent focuses on multistep tool execution. The distinction should not be reduced to a ranking of visual representation quality.
• vs ThinkGeo / Earth-Agent: This work trains its controller on validated trajectories rather than only providing an evaluation environment. Cross-benchmark results support partial transfer, subject to interface adaptation and metric-specific boundaries.
• vs ReAct / OpenThinkIMG: The thought-action-observation loop is not new here. The distinctive elements are geospatial data, unified GIS/spectral interfaces, and spatial-consistency validation. The transferable lesson is to add executable semantic constraints to domain tools, not merely change prompts.

Rating

• Novelty: 4/5. The contribution emphasizes systems and supervision-data integration, with targeted spatial validation, while the control loop and supervised-learning paradigm build on established ideas.
• Experimental Thoroughness: 3/5. Step-by-step, end-to-end, and cross-benchmark results are provided, but the available main paper lacks verifiable component ablations and fuller geographic extrapolation analysis.
• Writing Quality: 4/5. The examples are intuitive and tool/training boundaries are reasonably clear, although evaluation protocols require careful distinction.
• Value: 4/5. The work offers a practical training and evaluation approach for executable remote-sensing agents, with a remaining gap to reliable autonomous analysis.