RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios¶

Conference: ECCV 2026
arXiv: 2511.18011
Code: https://github.com/tsinghua-fib-lab/RoadBench
Area: Multimodal VLM / Spatial Reasoning / Autonomous Driving
Keywords: MLLM Evaluation, Fine-Grained Spatial Understanding, Road Marking Reasoning, Dual-Perspective BEV/FPV Benchmark, Urban Scenarios

TL;DR¶

RoadBench proposes the first fine-grained spatial understanding and reasoning benchmark in urban road scenarios with road markings as the core, consisting of 8 graded tasks with a total of 3,040 manually verified test cases across dual BEV (satellite) and FPV (car-mounted) perspectives. A systematic evaluation of 20 mainstream MLLMs reveals that existing models severely lack fine-grained spatial understanding capabilities—the best model achieves an F1 score of only 0.355 on BEV lane counting, and even falls short of a rule-based baseline utilizing traffic common sense on lane direction recognition. This exposes a massive capability gap in MLLMs between "seeing" and "reasoning".

Background & Motivation¶

MLLMs have made significant progress in general visual understanding and spatial reasoning. As a key application scenario, urban roads have seen several spatial understanding benchmarks emerge, such as CityBench, UrBench, and DriveBench. However, the evaluation granularity of these benchmarks is limited to the recognition of coarse-grained isolated objects such as buildings, vehicles, and traffic signs, leaving road markings (lane lines, turn arrows, etc.)—the fine-grained structural elements that actually organize urban spaces—almost entirely uncovered.

This is not an accidental oversight but is due to real technical barriers: First, road markings are slender and stretch across the entire image, demanding that MLLMs possess global-range fine-grained structural perception capabilities—not just locating objects inside local patches, but tracing lines of millimeter-scale width across the entire image scale. Second, there are strong mutual dependencies among road markings; the semantics of a single lane line depend on the holistic system formed with other markings (e.g., multi-lane turn rules). Isolated evaluation fails to capture this joint understanding requirement. Third, constructing high-quality annotated data is extremely costly, requiring professional transportation knowledge and making fully automated algorithmic generation unfeasible—explaining why this has not been done before.

The feasibility of this work stems from three key enabling factors: First, the foundational capabilities of MLLMs have evolved to a level where they can challenge such fine-grained tasks. Second, databases from leading map service providers (such as AutoNavi) provide high-precision ground-truth annotations (e.g., lane count, lane direction, and intersection locations) verified by online services. Third, the prevalence of satellite imagery and street-view data enables systematic evaluation across dual BEV and FPV perspectives.

Using road markings as an entry point, RoadBench designs 8 graded tasks across dual BEV and FPV perspectives, spanning from local perception (lane counting) to joint understanding (lane direction recognition), global reasoning (road network correction), and cross-view alignment. This covers four capability chains of MLLMs for fine-grained spatial elements: recognition, joint understanding, reasoning, and domain knowledge fusion.

Core Idea: Use road markings, the most fundamental yet neglected fine-grained element in urban spaces, as a "probe" to systematically diagnose the complete spatial intelligence of MLLMs from low-level perception to high-level reasoning in real and complex scenarios, exposing capability shortfalls masked by coarse-grained benchmarks.

Method¶

Overall Architecture¶

The core design philosophy of RoadBench is to decompose spatial understanding into a capability chain from local to global, progressively evaluating MLLM's fine-grained spatial intelligence across 8 tasks. The inputs consist of two types of urban scenario images: BEV satellite images (2,137 images from Google Maps, covering intersection and road segment scenarios) and FPV vehicle-mounted camera images (721 images from AutoNavi crowdsourced data, covering various lighting and occlusion conditions). All images are gathered from 5 cities in China with relatively consistent traffic rules.

The 8 tasks are divided into four levels based on the required spatial range: The local perception layer includes BEV/FPV lane counting and FPV road type classification, requiring models to extract local fine-grained information (counting lanes, determining main/auxiliary roads); the joint understanding layer includes BEV/FPV lane direction recognition, requiring models to comprehensively understand the holistic semantics of a group of road markings (e.g., assigning directions for multiple lanes); the global reasoning layer includes BEV road network correction, requiring models to reason about the spatial structure of intersections and segments based on global image context and output precise coordinates; and the cross-view reasoning layer includes cross-view versions of lane counting and direction recognition, requiring models to establish spatial correspondences and perform joint reasoning across geographically paired BEV-FPV image pairs. All 3,040 test cases were manually verified individually by 5 domain experts on the LabelU platform, where ambiguous samples were directly discarded rather than re-annotated, guaranteeing zero subjective bias in labels.

Key Designs¶

1. Graded Task System: Full Capability Chain Evaluation from "Lane Counting" to "Road Network Correction"

Existing urban scenario benchmarks often utilize flat task designs, where different tasks evaluate isolated capabilities in parallel, lacking progressive relations. The core design choice of RoadBench is to organize the 8 tasks into an explicit progression chain of difficulty and capability: the bottom-level lane counting only requires MLLMs to identify the number of slender lane lines within the area indicated by a reference line (pure perception); the mid-level lane direction recognition requires joint understanding of the combined semantics of multi-lane markings and the integration of traffic domain knowledge on top of that (perception + joint understanding + knowledge fusion); the higher-level road network correction requires models to output precise structured coordinates based on understanding road topology (perception + global reasoning + precise structured output); the top-level Cross-View tasks require establishing scene-level spatial correspondences across different perspectives (cross-view over alignment + fusion). This progressive design allows researchers to pinpoint the exact failure points of MLLM spatial capabilities—for example, if a model performs decently on FPV lane counting but drops drastically on the same task in BEV, it indicates that its bottleneck lies in global fine-grained perception rather than semantic understanding.

2. Dual-Perspective Complementary Design: Tension Field of BEV Topology and FPV Semantic Details

The BEV and FPV perspectives are naturally complimentary in spatial information distribution: BEV provides a complete bird's-eye view of road topology, where road markings are clear across the entire image scale but have limited resolution; FPV provides high-fidelity road semantic details (marking textures, traffic signs, environmental context) but suffers from restricted field-of-view and a lack of depth information. RoadBench sets both BEV and FPV versions for the same task (e.g., lane counting and direction recognition). This is not simply "testing again with a different view", but intentionally constructing an information-asymmetric tension field—the BEV task forces MLLMs to rely on global fine-grained structural analysis (as markings are very thin in BEV), while FPV tasks test whether models can infer global spatial relations from local high-resolution clues. Experimental results show that MLLM performance on FPV is systematically superior to BEV, a gap that itself reveals that the current models' genuine shortcoming lies in global fine-grained perception rather than semantic reasoning. Furthermore, the Cross-View tasks push this tension to the extreme: models must align two perspectives with entirely different information distributions, evaluating whether they truly "understand" the spatial structure of the scene rather than just performing pattern matching.

3. Three-Stage Data Construction Pipeline: Automated Matching Grounding + Manual Verification Safeguard + Direct Discarding of Ambiguities

Large-scale, high-quality annotated data is the biggest bottleneck for fine-grained benchmarks. RoadBench addresses this with a three-stage pipeline. Data Preparation Stage: Intersection locations are extracted from OpenStreetMap, and image validity and clarity are manually verified on Google Maps satellite imagery one by one to eliminate false intersections, severe occlusions, and unrecognizable samples. Meanwhile, during FPV image collection, challenging scenarios such as nighttime, backlight, and marking occlusions are actively included to ensure comprehensive difficulty coverage. Data Processing Stage: BEV bounding boxes and FPV capture coordinates are queried against the map service provider's database (AutoNavi) to automatically match and extract high-precision ground truths verified by online services, including lane counts, lane directions, and intersection locations. For the road network correction task, differences between the original OSM reference lines and the database ground truths are utilized to automatically construct "incorrect reference line + correct annotation" sample pairs. Quality Control Stage: 5 domain experts conduct a comprehensive manual review of all test cases on the LabelU platform. A critical principle is that ambiguous samples are directly discarded rather than manually re-annotated—this prevents subjective annotation biases from entering the benchmark, ensuring that all retained samples have undisputed labels. Furthermore, all data is de-identified (UUID randomized IDs, coordinates converted to pixel coordinates, human faces and license plates in FPV images manually blurred).

4. Task-Specific Multi-Dimensional Evaluation Metric System: From Classification Accuracy to Structured Output Fidelity

Different tasks place distinct requirements on MLLM outputs, and generic evaluation metrics can obscure key capability variations. RoadBench carefully selects metrics that best reflect the core challenges of each task: Lane counting, treated as a multi-class classification problem, reports both the category-imbalance-aware Weighted F1-Score and the RMSE, which directly reflects the deviation in counts (the two are complementary—F1 measures classification accuracy, while RMSE measures the practical impact of "how many lanes were miscounted"). Since lane direction recognition involves a multi-label combination for each lane (e.g., "straight + right turn"), Hamming Loss is adopted as the main metric (measuring average misjudgment rate per label), along with exact match accuracy (Acc) as a strict standard (requiring all lane directions to be completely correct to be considered right). For road network correction, as the output consists of intersection coordinate points and road segment polylines, RMSE with a maximum distance threshold is used to evaluate intersection point deviations, and Fréchet Distance is used to evaluate road segment polyline similarity. The key reason for selecting Fréchet Distance over Hausdorff Distance is that the former accounts for the directionality of the polylines; in road network correction, directional errors are as fatal as positional errors. All distance metrics are computed after normalizing coordinates to [0,1], and the THETA threshold is set to three levels {10%, 20%, 50%} to analyze model performance under varying precision requirements.

Key Experimental Results¶

Main Results¶

RoadBench evaluated 20 mainstream MLLMs, including open-source (LLaMA-3.2-Vision 11B/90B, Qwen2.5-VL 7B/32B/72B, Qwen3-VL 2B-235B, Gemma-3 12B/27B, GLM-4.5V) and closed-source (Gemini-2.5 Flash/Pro, GPT-5 Nano/Mini/5) series. The table below summarizes the comparison between the best-performing models and rule-based baselines across tasks.

Task	Metric	Best Closed-source Model	Best Open-source Model	Rule/Random Baseline
BEV Lane Counting	F1↑ / RMSE↓	GPT-5-Mini 0.355/1.162	GLM-4.5V 0.289/1.183	0.248/1.383
BEV Direction Recog.	HL↓ / Acc↑	GPT-5-Mini 0.103/0.662	Qwen3-VL-32B 0.188/0.515	0.149/0.587
BEV Road Net. Corr. (Inter.)	RMSE@20%↓	GPT-5-Nano 0.132	Qwen3-VL-235B 0.139	0.156(Do Nothing)
BEV Road Net. Corr. (Seg.)	FD@50%↓	GPT-5-Mini 0.235	Gemma-3-27B 0.232	0.379(Do Nothing)
FPV Lane Counting	F1↑ / RMSE↓	Gemini-2.5-Pro 0.522/0.802	Qwen3-VL-235B 0.369/1.034	0.284/1.430
FPV Direction Recog.	HL↓ / Acc↑	GPT-5 0.122/0.626	GLM-4.5V 0.145/0.571	0.127/0.605
FPV Road Type	Acc↑	Gemini-2.5-Pro 0.748	GLM-4.5V 0.570	0.458
CV Lane Counting	F1↑ / RMSE↓	GPT-5 0.400/0.878	Qwen3-VL-32B 0.298/1.185	0.252/1.395
CV Direction Recog.	HL↓ / Acc↑	GPT-5-Mini 0.121/0.588	GLM-4.5V 0.142/0.557	0.130/0.588

Core Conclusions: (1) None of the models achieve usable performance on any of the tasks—the best F1 on BEV lane counting is only 0.355, which is far from saturated. (2) On the three tasks of BEV direction recognition, FPV direction recognition, and CV direction recognition, the majority of MLLMs fall short of a rule-based lookup table method based on traffic common sense. (3) In BEV road network correction, the optimal values of intersection RMSE@20% and road segment FD@50% are in the same order of magnitude as the upper-limit thresholds, indicating that the coordinates output by the models suffer from severe offsets, making it fundamentally impossible to achieve precise structured spatial output. (4) FPV task performance is systematically superior to that of the corresponding BEV tasks, with the gap in F1 reaching up to 0.2+, indicating that current MLLM bottlenecks lie in global fine-grained perception under the BEV perspective.

Fine-Tuning Experiments¶

Model	BEV LC F1↑	BEV LD HL↓	BEV RNC RMSE@20%↓	FPV LC F1↑	FPV LD HL↓	FPV RTC Acc↑	CV LC F1↑	CV LD HL↓
Qwen3-VL-2B	0.186	0.313	0.154	0.210	0.286	0.539	0.152	0.268
Qwen3-VL-2B(tuned)	0.319	0.232	0.114	0.387	0.103	0.830	0.421	0.105
Qwen3-VL-4B(tuned)	0.339	0.230	0.107	0.517	0.100	0.800	0.535	0.096
Qwen3-VL-8B(tuned)	0.339	0.230	0.097	0.433	0.094	0.830	0.540	0.085

After fine-tuning on the RoadBench training set, the performance of the Qwen3-VL series models improved across the board: the smallest 2B(tuned) model jumped from 0.186 to 0.319 in BEV lane counting F1, surpassing GPT-5 (0.305); the FPV road type classification accuracy surged from 0.539 to 0.830, outperforming Gemini-2.5-Pro (0.748); the cross-view tasks also showed substantial improvements, demonstrating that these capabilities are not entirely unlearnable. However, a gap in overall performance still exists between the fine-tuned models and top closed-source models like GPT-5 and Gemini-2.5-Pro, indicating that model scale and pre-training pipelines remain critical advantages.

Key Findings¶

Fine-grained perception is the Achilles' heel of MLLMs: Under BEV, no model's F1 exceeds 0.36, with errors exceeding 1.16 lanes, indicating that the capability of MLLMs to trace millimeter-scale thin lines on a global scale is virtually zero. This is further corroborated in the ablation analysis—in FPV tasks, when road markings are occluded and models must rely on other cues like signs instead, the performances of GPT-5 and Gemini-2.5-Pro actually improve, suggesting that the visual perception capability of these powerful models is far weaker than their common-sense reasoning capability.
Rule-based baselines expose "pseudo-reasoning": In BEV/FPV direction recognition, a lookup rule based on traffic common sense (leftmost lane = left turn, rightmost lane = right turn, middle = straight) surprisingly outperforms all models except GPT-5-Mini without even looking at the images. This implies that the "reasoning" of current MLLMs on these tasks is largely guessing based on superficial visual features rather than a genuine understanding of spatial structure.
Cross-view alignment remains an open challenge: In Cross-View tasks, even the strongest models like GPT-5 and Gemini-2.5-Pro exhibit significant performance degradation. Most models fail completely on Cross-View direction recognition (with Gemini-2.5-Pro yielding a high HL of 0.282), indicating that the ability to establish unified spatial representations from two perspectives with entirely different information distributions has not yet emerged in existing MLLMs.
Parameter scale is not a panacea: GPT-5-Mini outperforms GPT-5 on most tasks, and Qwen2.5-VL-32B outperforms its 72B version on FPV direction recognition, demonstrating that the impact of vision-language modality fusion strategies and training data distribution on fine-grained spatial capabilities may outweigh parameter scale.
Prompt format preferences show significant model variance: In the ablation of reference line prompting methods for BEV tasks, GLM-4.5V exhibits a strong dependency on arrow/bar visual prompts (performance plummets when removed), whereas GPT-5-Mini is insensitive to different prompt formats, reflecting an intrinsic difference in their vision-text information integration mechanisms.

Highlights & Insights¶

"Falling short of the rule-based baseline" is the most powerful finding of this paper: Beating almost all MLLM performances on direction recognition with the simplest traffic common-sense lookup table provides a counter-intuitive yet highly convincing proof of MLLMs' lack of fine-grained spatial understanding compared to absolute scores. This experimental design approach—using an image-independent rule baseline to calibrate "how much the model's visual input actually contributes"—is highly recommended for all benchmark papers.
The "direct discard of ambiguities" quality control strategy: While typical benchmarks correct annotation errors when found, RoadBench opts to directly remove ambiguous samples. This seemingly conservative strategy actually guarantees two key properties: zero subjective bias in labels (no pollution by the annotator's personal judgment) and a high signal-to-noise ratio in evaluation signals (all retained samples can be correctly answered by human experts without dispute). This is particularly crucial for evaluating MLLMs that perform poorly to begin with—if even the labels are contentious, it is impossible to determine whether the model is wrong or the annotation is flawed.
The FPV > BEV performance asymmetry as a diagnostic tool: Instead of stopping at reporting the superficial conclusion that "FPV is better than BEV", the authors interpret this as a capability diagnostic signal—since the semantic understanding capabilities of the same model across FPV and BEV should be identical, the significant inferiority on BEV can only be attributed to a deficit in global fine-grained visual perception. This "control variable"-style task design allows the benchmark itself to serve as an analytical tool.
The profound rationale behind choosing Fréchet Distance: In road network correction, Fréchet Distance is used instead of the more common Hausdorff Distance to evaluate polyline similarity because Fréchet accounts for the directionality of the polylines—two polylines that are similar in shape but run in opposite directions have a small Hausdorff distance but a large Fréchet distance. In road network correction, direction errors (such as misinterpreting a two-way road as a one-way road) are as fatal as positional errors; this choice of metric reflects a deep understanding of the task's essence.

Limitations & Future Work¶

Limitations in geography and traffic rules: The data is sourced from only 5 cities in China, where traffic rules (e.g., driving on the right, specific Chinese lane marking standards) are relatively uniform. Generalization to countries with left-hand driving or different marking systems remains unverified, which is an essential evaluation dimension before practical deployment.
Lack of a human baseline: The authors argue that since the quality control stage ensured humans can complete all tasks completely correctly, a human baseline is unnecessary. However, this evades a key question—how much professional training is required? Can laypeople complete it? What is the speed and accuracy of human completion? A truly comprehensive benchmark should quantify "human difficulty" as a reference.
Small sample sizes in certain splits: Test cases for FPV direction recognition and Cross-View tasks are relatively small in number (constrained by the high acquisition cost of geographically paired BEV-FPV data). Consequently, minor performance differences on these tasks should not be over-interpreted as definitive model rankings. Although the authors added the mean and standard deviation of three repeated evaluations in the appendix to alleviate this issue, expanding the scale of these splits remains fundamentally necessary.
Evaluating a coupling of "instruction following + perception + reasoning": Under the current setup, if a model understands the image but fails to output in the YAML format, it is marked as an error. Although a 6-time retry mechanism is implemented, the interference of output format parsing failures on the evaluation results cannot be entirely ruled out.
Extensible directions: Broader evaluations ranging from road markings to other fine-grained urban elements (e.g., curbs, crosswalks, barriers); introducing agent frameworks to evaluate MLLMs' spatial reasoning capabilities in multi-step interactions (rather than single-turn feedforward); and adding a temporal dimension (sequential FPV video frames) to evaluate dynamic spatial understanding.

vs CityBench: CityBench evaluates MLLMs' comprehensive capabilities in urban perception, planning, and decision-making, with granularity remaining at the building/regional level and mostly featuring Q&A on isolated objects. RoadBench refines the granularity down to the road marking level and emphasizes inter-element dependencies and a progressive capability evaluation chain. The relationship between the two is complementary rather than competitive.
vs UrBench: UrBench also conducts BEV+FPV cross-view evaluation, but its finest-grained elements are traffic signs (typically tens of centimeters in size), and its cross-view tasks are primarily matching-based. RoadBench's road markings (a few centimeters wide) are an order of magnitude finer in spatial granularity than traffic signs, and its cross-view tasks demand joint reasoning rather than simple matching. The granularity leap from UrBench to RoadBench reflects the deepening of the community's evaluation of MLLM spatial capabilities.
vs DriveBench/NuPlanQA: These two benchmarks focus on MLLM reliability evaluation in autonomous driving scenarios, representing a complementary dimension to RoadBench's "fine-grained spatial understanding diagnostic"—the former concerns "whether the model will fail in the autonomous driving pipeline", whereas the latter concerns "how strong the model's underlying spatial perception capability is". A practical autonomous driving system requires passing both hurdles.
Methodological Inspiration: The "rule-based baseline exposing pseudo-reasoning" paradigm from RoadBench is transferable to any benchmark design involving common-sense reasoning—the margin by which your model outperforms an input-independent simple rule is the actual incremental capability the model learns from visual/textual inputs. This design approach is highly effective in combating score saturation in benchmark domains.

Rating¶

Novelty: ⭐⭐⭐⭐☆ A fine-grained spatial understanding benchmark using road markings as an entry point clearly fills a blank. The 8-task graded system and dual-perspective design have clear motivations and differentiated positioning.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Fully comprehensive evaluation of 20 mainstream MLLMs, fine-tuning experiments, prompt ablations, environmental condition analysis, 3 repeated evaluations, and thorough data in the appendix—the experimental standards for a benchmark paper are thoroughly met.
Writing Quality: ⭐⭐⭐⭐☆ The rationales behind task definitions and chosen evaluation metrics are clearly articulated, and core findings (failing against rule baselines, FPV > BEV asymmetry) are consistently reinforced; however, descriptions of some tasks are brief (e.g., the details of prompt designs for Cross-View are compressed into the appendix).
Value: ⭐⭐⭐⭐☆ Holds clear diagnostic value for MLLM spatial understanding research—it is not "just another benchmark for leaderboard chasing", but provides concrete evidence revealing the systematic deficiencies of current models in fine-grained perception; it also offers direct guidance for MLLM applications in autonomous driving and HD mapping.