OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/SIGMME/OmniMapBench
Area: Vision-Language Reasoning
Keywords: map understanding, visual question answering, multi-step spatial reasoning, Visual Dependency Index, textual shortcuts

TL;DR

OmniMapBench tests visually grounded reasoning with 2,096 manually verified questions across nine map categories and controlled text-description comparisons; the best of 25 evaluated models achieves a final score of 75.03%, while the benchmark retains a Visual Dependency Index of 0.338 under a 1,024-token description budget.

Background & Motivation

Improving document question-answering scores do not necessarily establish that vision-language models (VLMs) can perform complex visual reasoning. Many forms, reports, and charts can first be converted into structured text through OCR, layout analysis, or chart-to-table conversion, then solved by a large language model. For example, preserving the rows and columns of a statistical table often turns a question about a group's percentage into textual lookup. This does not diminish document recognition, but it obscures a more specific question: does successful answering depend on visual structure or on information already available as text?

Maps offer a different test because answers often depend on route continuity, relative directions, region boundaries, and associations between legends and symbols. Knowing which station names appear does not establish whether two stations are directly connected; listing buildings may not recover left-right relationships along a journey. Earlier map benchmarks often focus on transit or choropleth maps, leaving substantial differences among tourist diagrams, evacuation plans, topographic maps, and game maps unexplored. The paper therefore broadens both map categories and reasoning levels instead of merely adding more questions about one map style.

A second important choice is to hide the question from the image describer, preventing it from selectively recording exactly the local relation needed for the answer. If the complete image supports answering but a general description from the same model does not, the information substitution incurs a measurable loss. Whether this loss disappears with a larger description budget reveals more than a single no-image test about information omitted by general textual summaries. Core Idea: evaluate model capabilities with diverse, hierarchically labeled map questions, then replace images with question-agnostic descriptions to measure visual dependency under an explicit budget.

Method

Overall Architecture

The contribution is a dataset and evaluation protocol, not a new map recognition network or an OmniMapBench-specific answering model. Dataset construction proceeds through map acquisition and filtering, hierarchical question annotation, and independent cross-validation, producing images, questions, choices, and ground-truth answers. Standard evaluation presents the map image and question to a tested VLM, uses chain-of-thought (CoT) prompting, and checks whether the final answer matches exactly. Breaking down scores by capability level and map category helps distinguish basic perception difficulties from complex relational inference.

The visual dependency experiment instead fixes one reference model and compares direct visual answering with a describe-then-answer pathway. Its intermediate representation is a token-limited image description generated without the question, not a human-authored ground-truth topology of the map. The answering stage receives only this description and the question, with no further image access; the experiment measures how much answer-relevant information this description protocol preserves. Outputs include standard scores, the Visual Dependency Index (VDI), and the Descriptive Saturation Index (DSI), rather than training losses or reconstructed maps.

Key Designs

1. Diverse map acquisition: preserve stylistic variation and fine visual detail

The authors initially collect more than 5,000 map images from the web, using GPT-5 to expand seed search terms across nine predefined categories. Automatic scraping is supplemented with manual collection from government portals and social platforms for transit and tourist maps with particular clarity and style requirements. They remove highly similar images, images with insufficient resolution or clarity, and content involving politically sensitive regions, retaining approximately 2,000 images for annotation. The final questions reference 1,603 images; this count belongs to a different stage from the pre-annotation image pool and should not be conflated with it. The nine categories are Indoor Navigation, Education & History, Engineering & Facilities, Transportation, Tourism, Topography & Remote Sensing, Environment & Nature, Socioeconomic, and Game. Indoor Navigation contains 294 images, whereas Game contains only 39, so visual diversity does not imply equal category sizes.

Images retain their native resolutions, ranging from 233 × 464 to 11,811 × 9,442 pixels, with a mean of 2,055 × 1,705. Small text, dense symbols, and connections are therefore not necessarily discarded by uniform downsampling during dataset preparation, preserving realistic map complexity. However, retaining the source image describes dataset storage, not a guarantee that all tested models process it at the same internal resolution. Interpreting results consequently requires attention to visual encoding, tiling, and resolution policies, not just language model size. This is an interpretive boundary for the evaluation, not a resolution-controlled experiment completed by the paper.

2. Hierarchical annotation and independent review: make questions map-dependent and objectively scorable

Professional annotators write questions that can be answered using the map alone, without geographic knowledge or external retrieval. Single-choice, multiple-choice, and ordering formats constrain answers to an explicit space, reducing ambiguity arising from alternative free-text formulations. Questions are also labeled as L1 perception, L2 single-step spatial reasoning, or L3 multi-step relational reasoning, corresponding to extracting local information, judging direct spatial relations, and composing multiple relations. For instance, tracing the regions encountered along a route requires maintaining path order, whereas identifying a symbol above a building entrance primarily requires local perception. These levels are dataset labels, not a training curriculum or three model modules that every question must traverse.

An initial 3,386 questions are filtered and balanced down to 2,882, then reviewed by two independent teams. Reviewers did not participate in the initial annotation of the data they inspect; an item is discarded if either team finds ambiguity, an incorrect answer, or an inappropriate capability label. The final 2,096 questions comprise 1,461 single-choice, 269 multiple-choice, and 366 ordering questions, representing 69.70%, 12.83%, and 17.46%. The capability distribution is 441 L1, 802 L2, and 853 L3 questions, or 21.04%, 38.26%, and 40.70% (Table 1, page 7). There are 1,274 English and 822 Chinese questions, representing 60.78% and 39.22%; this does not establish paired English and Chinese versions of each question. The review procedure targets answer reliability and task definition, but the main text provides no inter-annotator agreement coefficient, so it cannot establish complete elimination of subjective bias.

3. Budgeted textual substitution and VDI: measure answer-relevant information omitted by general descriptions

The reference model is fixed to gpt-4.1-2025-04-14, which performs direct answering, image description, and description-based answering. First, its full-input accuracy is measured with images and questions; descriptions are then generated under maximum budgets of 64, 128, 256, 512, and 1,024 tokens. The description prompt requests detail and permits code for charts or tables, so the protocol does not deliberately prohibit all structured representations. The crucial restriction is that the describer cannot see the question; otherwise, it could select question-specific evidence, changing the experiment from general textual substitution to question-driven extraction. The image is then removed, and the description and original question are submitted to the model to obtain text-path exact-match accuracy.

VDI divides the difference between full-input and text-path accuracy by full-input accuracy. The following expression restates line 12 of Algorithm 1 on page 12; the typeset equation on page 10 is corrupted in the extraction, but the algorithm and verbal definition agree.

\[ \mathrm{VDI}(k_{\max})=\frac{\mathrm{Acc}_{\mathrm{full}}-\mathrm{Acc}_{\mathrm{text}}(k_{\max})}{\mathrm{Acc}_{\mathrm{full}}}. \]

The denominator must be nonzero; 0.338 means a 33.8% relative accuracy loss against the full-image pathway, not an absolute decline of 33.8 percentage points. A smaller VDI indicates that the description preserves most answer-relevant information under this protocol, whereas a larger value indicates substantial residual loss. The index nevertheless depends on the reference model, description prompt, and budget; it is not an implementation-independent constant of visual indispensability. The definition also imposes no clipping: a description pathway that outperforms full input could theoretically yield a negative value, so the paper's approximate 0-to-1 range is not a strict bound.

4. DSI and stratified controls: distinguish description length, linguistic guessing, and multi-step difficulty

A maximum token budget does not guarantee that the model will produce a description of that length, so the authors also record each sample's actual description length. DSI is the sample average of actual length divided by the allowed maximum, providing a complementary view of whether descriptions consistently exhaust the budget. The following expression follows the averaging and normalization in line 13 of Algorithm 1 on page 12.

\[ \mathrm{DSI}(k_{\max})=\frac{1}{N}\sum_{i=1}^{N}\frac{k_{\mathrm{actual},i}}{k_{\max}}. \]

A high DSI only indicates descriptions approaching the length limit; it does not establish description accuracy or independently demonstrate visual necessity. Reading DSI alongside VDI helps distinguish cases where a short description suffices from cases where a relatively long description still omits answer-relevant information. The authors additionally provide a fully blind control: models receive only questions and choices, without either images or image descriptions. Unlike the text pathway in VDI, this tests whether linguistic priors, world knowledge, and answer-choice bias alone can support high scores. Finally, the L1, L2, and L3 breakdown reveals whether errors concentrate in multi-step relation composition instead of inferring their causes from the final score alone. All these interventions occur during evaluation; the main text introduces no dedicated training objective, fine-tuning procedure, or new model loss.

Key Experimental Results

Main Results

Table 4 (page 13) evaluates 25 models with CoT prompting and requires exact final-answer matches for single-choice, multiple-choice, and ordering questions. The table below selects representative models; all values are percentages, and Final Score is preserved as reported. Detailed prompts and decoding settings are deferred to the supplement, which is absent from the supplied full-text cache, so temperature and output limits are not reconstructed here.

Model	L1 Perception	L2 Single-step	L3 Multi-step	Final Score
Gemini-3.1-Pro	88.24	81.53	62.88	75.03
Qwen3.5-397B-A17B	83.89	79.01	60.37	72.22
Qwen3.5-27B	82.93	77.10	53.97	68.62
Kimi-K2.5	74.77	74.80	56.71	67.64
Gemini-2.5-Pro	73.70	70.70	46.31	61.40
GPT-5	63.10	61.47	48.53	56.54
GLM-4.5V	66.67	54.24	35.80	49.33
GPT-4.1	48.52	45.14	37.87	42.88
GPT-4o	40.09	31.80	26.73	31.47

Gemini-3.1-Pro leads Qwen3.5-397B-A17B by 2.81 percentage points in Final Score; this does not establish that proprietary models consistently outperform open-source models. Its accuracy decreases from 88.24% on L1 to 62.88% on L3, a difference of 25.36 percentage points, leaving multi-step questions a substantial weakness. These are different question subsets, not a controlled experiment that progressively increases the reasoning steps of the same question.

Ablation Study

Table 2 (page 8) uses the fixed reference model gpt-4.1-2025-04-14 to compare full-image answering with question-agnostic description-based answering. The table below selects two description limits; VDI and DSI are dimensionless and retain the source's three decimal places.

Benchmark	VDI, 512 tokens	DSI, 512 tokens	VDI, 1,024 tokens	DSI, 1,024 tokens
OmniMapBench	0.474	0.915	0.338	0.629
AI2D	0.035	0.597	0.019	0.336
ChartQA	0.147	0.698	0.143	0.394
DocVQA	0.122	0.795	0.076	0.516
InfoVQA	0.155	0.961	0.049	0.762
MathVista	0.126	0.478	0.051	0.269
MMMU-Pro	0.257	0.459	0.117	0.273
OCRBench	0.044	0.244	0.043	0.124

At 1,024 tokens, OmniMapBench has the highest VDI among all 11 benchmarks in the complete source table; its DSI is below InfoVQA's and is not the highest. Longer descriptions do help: OmniMapBench's VDI decreases from 0.474 at 512 tokens to 0.338, but does not approach zero. Page 15 separately reports that the mean accuracy of three representative models falls from 58.87% to 23.35% without images, an absolute decline of 35.52 percentage points. Per-model blind results and detailed reference-model replacement ablations are deferred to the supplement, so individual model declines cannot be inferred from this average.

Key Findings

• Multi-step relational reasoning is harder than local perception, although exact matching also makes a local ordering error invalidate an entire answer.
• High VDI and low blind accuracy are complementary: the former measures information lost through descriptive substitution, whereas the latter measures guessing without visual evidence.
• Pages 14–15 describe Tourism as generally easier and Topography & Remote Sensing as harder; the radar chart does not provide reliably extractable category values, so numerical scores are not estimated.

Highlights & Insights

• Question-agnostic description is an important control against hidden leakage in which the describer has already located the answer for the answering model. It more closely evaluates a general document representation that could be reused for multiple questions.
• A description-budget curve is more informative than a single blind score because it shows whether visual dependency persists as textual capacity increases. DSI additionally distinguishes allowed output length from actual output length.
• Diverse map styles expand evaluation beyond station-name recognition into symbols, connectivity, and ordered traversal. A transferable benchmark design principle is to vary both content types and relational complexity rather than only increasing question count.

Limitations & Future Work

• VDI is affected by the describer's perception, summarization choices, and text-answering ability; it does not prove that every textual representation must fail. Stronger structured map representations, question-driven retrieval, and larger description budgets are useful additional comparisons, not approaches already shown to fail here.
• Categories are imbalanced, and L3 accounts for 40.70% of questions, making the final score sensitive to question composition. The main text reports no confidence intervals, inter-annotator agreement coefficient, or paired cross-lingual control, so small ranking differences and language effects need cautious interpretation.
• The supplement is absent from the supplied cache, preventing verification of complete reference-model ablations, per-model blind results, and detailed decoding settings. Some extracted equations are corrupted; this note retains only the VDI and DSI definitions independently supported by Algorithm 1.
• More targeted error analysis could separate connection tracing, legend matching, and viewpoint-dependent directions. This is a reader suggestion for locating the boundary between perception and reasoning, not an ablation result established by the paper.

Related Work & Insights

• Compared with DocVQA and ChartQA: these often permit compact textual representations through OCR or tabular conversion, whereas OmniMapBench emphasizes map topology and symbolic relationships. VDI supplies an empirical comparison under a common reference model and budget, not a rejection of those benchmarks' broader value.
• Compared with ReasonMap and MapQA: ReasonMap focuses on transit maps, while MapQA works address choropleth maps and OpenStreetMap; OmniMapBench covers nine map categories. Differences in format, content, and sample size prevent interpreting dataset scale as a direct difficulty ranking.
• Implications for visual tool reasoning: local zooming, explicit route tracing, and structured relation storage may help with multi-step map questions. The paper supplies a benchmark for testing these directions, but does not establish gains from a particular tool-use method.

Rating

• Novelty: 4/5. Diverse maps and question-agnostic textual substitution define a clear evaluation problem; VDI itself is a simple normalized difference.
• Experimental Thoroughness: 4/5. Includes 25 models, dependency analysis across 11 benchmarks, and a blind control, but the available material lacks supplementary details.
• Writing Quality: 4/5. Dataset stages and evaluation goals are clear; claims of resistance to textualization still require model- and budget-specific interpretation.
• Value: 4/5. Useful for evaluating grounded visual relationships and identifying apparent progress obtained through textual shortcuts.