FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models¶

Conference: ICLR2026
arXiv: 2512.08016
Code: knowledge-computing/FRIEDA
Area: Multimodal VLM
Keywords: cartographic reasoning, map VQA, spatial relations, multi-image reasoning, benchmark

TL;DR¶

The FRIEDA benchmark is proposed to systematically evaluate the multi-step, cross-map cartographic reasoning capabilities of large vision-language models. Results show the strongest model, Gemini-2.5-Pro, achieves only 38.20% accuracy, significantly lower than the human performance of 84.87%.

Background & Motivation¶

Cartographic reasoning is a core human cognitive ability involving the integrated understanding of legends, scales, compasses, map text, and geometric features, which is indispensable in real-world scenarios like urban planning and disaster response.
Existing LVLM research often treats maps as special cases of charts, ignoring map-specific symbolic grammar and spatial relation reasoning requirements.
Current map VQA benchmarks have significant limitations: (1) most cover only a subset of spatial relations (e.g., navigation or entity recognition only); (2) limited map styles (mostly choropleth or web basemaps); (3) lack of cross-map reasoning; (4) absence of in-document map retrieval scenarios.
Consequently, current benchmarks fail to comprehensively measure whether LVLMs possess human-level map reading capabilities.

Core Problem¶

How to design a cartographic reasoning benchmark that covers all three types of spatial relations (topological, metric, directional), requires multi-step reasoning and cross-map integration, and aligns with real-world document usage scenarios?

Method¶

Overall Architecture¶

FRIEDA is an open-ended VQA benchmark for cartographic reasoning. For each question, given one or more maps from a real document, the model must first interpret cartographic symbols such as the legend, map scale, and compass, then integrate evidence across maps to complete multi-step spatial reasoning, and finally provide a free-text answer. The benchmark consists of three components: a "four-dimensional task design" to define required skills, "expert-curated data construction" to extract and filter questions from real documents, and an "answer-type adaptive evaluation protocol" to fairly score open-ended responses. The final dataset comprises 500 high-quality questions filtered by PhD-level consensus, with nearly 60% requiring joint reasoning across multiple maps.

Key Designs¶

1. Four-Dimensional Task Design: Decomposing Human Map Reading Skills

Typical map VQA only covers specific spatial subsets, failing to measure human-level capability. FRIEDA aligns with GIS (Geographic Information System) literature to decompose skills into four orthogonal dimensions: The spatial relation dimension covers three categories and six relations—topological (border, equal, intersect, within, based on the 9-Intersection model), metric (distance, requiring scale-based conversion), and directional (orientation, requiring compass interpretation). The map element interpretation dimension requires understanding the semantics of map text, legends, map scales, and compasses. The cross-map reasoning dimension requires aligning symbols, labels, and scales shared across multiple maps. The contextual dimension requires models to retrieve the relevant map from multiple images in a document before answering.

2. Expert-Curated Data Construction: Mitigating Bias via Multi-Level Consensus

To ensure the benchmark tests visual reasoning rather than common knowledge, a multi-stage pipeline was implemented. Maps were collected from government reports, environmental impact statements, and geological surveys across 32 countries with diverse styles. Candidate questions were generated by GPT-4/GPT-o3 and filtered to ensure they could not be answered without the map or via search engines. GIS experts verified answers and resolved ambiguities. Finally, 11 PhD researchers performed multiple rounds of annotation, retaining only questions where \(\ge 2/3\) of annotators reached a consensus—removing 61 ambiguous questions.

Item	Quantity
Total Questions	500
Source Documents	210
Total Maps	17,030
Single-map Questions	202 (40.4%)
Multi-map Questions	298 (59.6%)
Questions requiring Legend	417 (83.4%)
Avg. Maps in Contextual set	9.5

3. Answer-Type Adaptive Evaluation Protocol: Fair Scoring for Open-Ended Answers

Since answers are free-text, literal matching is unsuitable. FRIEDA uses three sets of rules based on answer types. For text-based answers (topological/semantic labels), Mistral Small 3.1 serves as the LLM-as-Judge for semantic matching, tolerating equivalent expressions (e.g., "Cypress Creek" vs. "Cypress"). For distance-based answers, unit-aware parsing is followed by calculating the Mean Absolute Percentage Error (MAPE), where \(\text{MAPE} \le 20\%\) is considered correct. For directional answers, adjacent cardinal direction tolerances are allowed (e.g., NW or NE is accepted if the gold standard is North), avoiding excessive strictness on compass readings.

Key Experimental Results¶

Main Results¶

Model	Accuracy
Human Average	84.87%
Gemini-2.5-Pro	38.20%
GPT-5-Think	37.20%
Claude-Sonnet-4	31.60%
Qwen2.5-VL-72B (Best Open Source)	25.60%
Ovis2.5-9B-Think	25.80%

Spatial Relation Analysis¶

Orientation is the category where models perform best: Gemini-2.5-Pro reaches 71.59%.
Distance is the most difficult: the best model achieves only 27.47% (GPT-5-Think), and human performance is also lower (78.28%).
For equal relations, GPT-5-Think (44.44%) significantly outperforms Gemini-2.5-Pro (33.33%), demonstrating an advantage in multi-map reasoning.
Claude-Sonnet-4 performs best on distance questions, showing proficiency in scale interpretation.

Key Findings¶

Minimal accuracy difference between "direct" and "contextual" settings (88.03% question-level consistency) indicates the bottleneck is cartographic reasoning itself, not map retrieval.
Model size does not show a strong positive correlation with performance; training data and reasoning mechanisms are more critical.
Enabling "Think" mode for Ovis2.5-9B provides a ~5% improvement, primarily in orientation and multi-map alignment.

Error Analysis (Gemini-2.5-Pro)¶

Error Type	Proportion
Legend Misinterpretation (color/symbol mapping)	25.61%
Cross-map Interpretation Failure	23.78%
Spatial Relation Semantic Confusion	16.46%
Scale/Metric Error	9.76%
Map Text Selection Error	8.93%
Counting Error	6.71%

Highlights & Insights¶

Comprehensive Spatial Coverage: This is the first map VQA benchmark to systematically cover all three categories and six types of spatial relations.
Cross-Map Reasoning: 59.6% of questions require joint reasoning across multiple maps, filling a gap in evaluating multi-image integration in cartography.
Real Map Diversity: Maps from 210 documents across 32 countries cover six domains, avoiding the simplification bias of synthetic maps.
Strict Quality Control: Expert curation combined with 11 PhD annotators and \(\ge 2/3\) consensus filtering ensures high data quality.
Dual-Mode Evaluation: The direct and contextual settings successfully decouple reasoning ability from retrieval ability.

Limitations & Future Work¶

The dataset only includes documents with Latin characters, failing to cover maps in other languages like Chinese or Arabic.
The scale of 500 questions is relatively limited, and sample sizes for some spatial relation subcategories are unbalanced.
There is currently no evaluation of post-fine-tuning performance to judge if domain adaptation can significantly improve results.
Reliability of the LLM-as-Judge depends on specific models and may introduce bias.
The impact of chain-of-thought prompting or tool-augmentation (e.g., GIS APIs) has not been explored.

Dimension	MapQA/MapWise	MapEval	FRIEDA (Ours)
Map Type	Primarily Choropleth	Web Basemaps	Diverse Real Documents
Spatial Relations	N/A	Partial	All 3 categories / 6 types
Multi-map Reasoning	No	No	Yes (59.6%)
Document Context	No	No	Yes (contextual setting)
Answer Format	Multiple Choice	Multiple Choice / Short Answer	Open-ended

Unlike natural image spatial reasoning works like SpatialVLM, FRIEDA focuses on map-specific symbolic systems (legends, scales, compasses), evaluating symbol-semantic mapping rather than natural scene spatial perception.

Rating¶

Novelty: ⭐⭐⭐⭐ — First comprehensive real-world map reasoning benchmark covering multiple spatial categories.
Experimental Thoroughness: ⭐⭐⭐⭐ — Includes 11 models, human baselines, and fine-grained error analysis.
Writing Quality: ⭐⭐⭐⭐ — Clear task definitions with a strong link between GIS theory and LVLM evaluation.
Value: ⭐⭐⭐⭐ — Fills a critical gap in evaluation, meaningful for advancing spatial intelligence in LVLMs.