Skip to content

MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

Conference: ECCV 2026
Paper: ECCV page / project page
Code: https://github.com/Tajamul21/MedSPOT
Area: Medical Imaging
Keywords: Clinical GUI grounding / Sequential grounding benchmark / Multimodal LLM / Early-termination protocol / Failure taxonomy

TL;DR

MedSPOT is the first workflow-aware sequential grounding benchmark for clinical software interfaces: it records 216 interactive tasks with 597 annotated keyframes across 10 real medical imaging platforms, recasts GUI grounding as a 2-3 step state-dependent sequence of click decisions, and evaluates 16 MLLMs under a strict first-error-terminates protocol with a six-class failure taxonomy — the best model, GUI-Actor, reaches only 43.5% task completion accuracy, while human annotators reach 82.22%.

Background & Motivation

GUI grounding maps a natural-language instruction onto a precise clickable element in an interface. The field has moved quickly: ScreenSpot spreads evaluation across mobile, desktop and web interfaces, ScreenSpot-Pro pushes it to high-resolution professional applications and exposes severe degradation on small dense widgets; Mind2Web and OmniACT add multi-step browsing tasks, while OSWorld-G and AssistGUI move evaluation into real desktop environments. On the model side, UI-TARS folds perception, unified action modeling and System-2 reasoning into one end-to-end agent, and GUI-Actor performs coordinate-free grounding with a dedicated <ACTOR> token. Yet all these benchmarks share one assumption: they treat grounding as an isolated prediction problem — a screenshot plus an instruction, scored on whether the click lands correctly. Even the ones that support multi-step interaction score each step largely independently, so a wrong step does not affect the rest of the evaluation.

That assumption collapses in clinical software. Diagnostic viewers, treatment-planning systems and health-record dashboards share a distinctive structure: dense toolbars, deep menu hierarchies, highly specialized terminology, and icons that are semantically distinct but visually near-identical. More importantly, a real clinical procedure is itself a chain of causally dependent operations — a DICOM series can only be navigated after it is loaded, a study can only be deleted or exported after it is selected. A grounding error therefore is not merely "this step was wrong": it changes the next interface state and pushes every subsequent step off the intended trajectory, and in a safety-critical medical setting the cost of that early error propagating is far higher than a few extra clicks. Evaluations, however, are essentially silent on this. Existing GUI benchmarks are uniformly aimed at general-purpose software, while medical multimodal benchmarks (GMAI-MMBench, OmniMedVQA, MedSG-Bench) and medical visual grounding work such as VividMed evaluate static-image reasoning and localization and never touch interactive workflows inside software. General GUI benchmarks and medical benchmarks are each mature; the intersection — sequential grounding on safety-critical clinical interfaces — has been left empty.

The timing for filling that gap is ripe. Open-source medical imaging software — Orthanc, Weasis, 3D Slicer, ITK-SNAP, MITK, RadiAnt and others — is now widespread enough that complete operating procedures can be recorded in real products rather than faked with synthetic screenshots, and normalized-coordinate output from MLLMs is already the de facto grounding paradigm, so the model-side comparison set exists. The paper's design choices are correspondingly restrained: no synthetic interfaces, no invented tasks, but rather cutting existing real clinical operation videos into scorable keyframe sequences and then using an unforgiving evaluation rule to actually compress "sequence" into the score. Core idea: redefine GUI grounding from single-frame coordinate prediction into a state-dependent sequence of spatial decisions, and make both error propagation and systematic misunderstanding of interface structure explicitly measurable through a first-error-terminates protocol and a six-class failure taxonomy.

Method

Overall Architecture

What MedSPOT contributes is not a new network but a construction-and-evaluation apparatus that turns real clinical operations into scorable sequential grounding tasks. The input is GUI interaction video from medical software; the output is a task benchmark with step-level annotations plus a scoring scheme that characterizes error propagation. The pipeline has four stages: first formalize sequential grounding (a task is a chain of mutually dependent frame-instruction-action triples), then run the annotation pipeline that cuts raw screen recordings into keyframes and labels each frame with a semantic target and a normalized bounding box, then at evaluation time apply a first-error-terminates protocol that measures the longest correct prefix, and finally archive each step's failure into six interface-structure error classes for diagnosis.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Clinical software interaction video<br/>10 platforms · CT/MRI/PET/X-ray/US"] --> B["Sequential grounding formulation<br/>2–3 state-dependent click decisions"]
    B --> C["Clinical GUI data construction<br/>keyframe → annotation → task"]
    C --> D["First-error-terminates protocol<br/>longest correct prefix"]
    D -->|all steps correct| E["Task completion TCA"]
    D -->|first error terminates| F["Six-class failure taxonomy<br/>record failure type"]
    E --> G["SHR / S1A / WPS aggregation"]
    F --> G

Key Designs

1. Sequential grounding formulation: rewriting a click prediction as a state-dependent decision sequence

Most grounding datasets define a sample as an independent "instruction + screenshot → coordinate" pair; MedSPOT replaces this with a temporal structure. The dataset is organized per software platform, \(D = \bigcup_{s \in S} D_s\), where each platform \(s\) holds \(N_s\) task workflows \(D_s = \{T_i\}_{i=1}^{N_s}\), and a task \(T\) is a finite ordered sequence of interaction steps \(T = \{(I_t, s_t, A_t)\}_{t=1}^{T}\): \(I_t\) is the GUI frame at step \(t\), \(s_t\) is the natural-language instruction, and \(A_t\) is the corresponding ground-truth action. Each action is further decomposed into three parts, \(A_t = (a_t, y_t, B_t)\) — \(a_t \in \mathcal{A} = \{\text{click}\}\) is the action type, \(y_t\) is the semantic description of the target, and \(B_t = (x, y, w, h) \in [0, 100]^4\) is the bounding box in normalized percentage coordinates. Annotating the semantic target description alongside the bounding box is deliberate: it forces the model to learn two things jointly, localizing the region from instruction and image \(P(\text{region} \mid s_t, I_t)\) and recognizing what widget that region is \(P(y_t \mid I_t)\). The latter is exactly what separates "clicked the right place" from "clicked the right icon."

What truly separates MedSPOT from earlier multi-step benchmarks is the state dependency between steps: the next interface state depends on the action just executed, \(P(I_{t+1} \mid I_t, a_t)\), so a grounding error at step \(t\) strips the downstream actions of their meaning. This constraint is not rhetorical — it directly determines that the evaluation rule must be sensitive to early errors, and it is precisely what "workflow-aware" means here: not merely permitting multiple steps, but acknowledging that every step changes the observation conditions of the next frame.

2. Clinical GUI data construction and annotation pipeline: cutting scorable keyframes out of real interaction video

All data comes from real software rather than synthetic interfaces. The authors record complete interaction workflows on 10 open-source medical imaging platforms covering five modalities (CT, MRI, PET, X-ray, ultrasound) and group the platforms into three interface families: DICOM/PACS viewers (109 tasks), segmentation and research tools (70 tasks), and web-based viewers (37 tasks). In scale, the benchmark holds 216 task videos and 597 annotated keyframes, with an average of 2-3 interdependent steps per task. By function, view/display operations dominate (81 tasks), followed by import/load (32), export/save (24), navigate/zoom (21), annotate/measure (20), tools/adjust (19), and settings/configuration (19). This distribution is itself informative about clinical workflows — viewing occupies far more of a reader's time than manipulation, so VQA-style static-image question answering never reaches these operations.

Annotation proceeds in three stages. Step extraction: raw recordings \(V_i = \{F_1, \dots, F_n\}\) contain many meaningless transition frames, so only a minimal ordered subset of decision frames is retained — each frame must correspond to a meaningful interaction decision point, and adjacent frames must genuinely exhibit a visible interface state transition (\(F_t \neq F_{t+1}\)) — yielding a compact causally consistent sequence. Frame-level annotation: each frame is labeled in Label Studio with \(A_t = (I_t, s_t, y_t, B_t, a_t)\), i.e. the instruction, the semantic description of the target (for instance "DICOM study list"), the bounding box and the action type; annotators first draw the target region, then write the instruction and semantic target, and finally pass a quality check against the source video. Task construction: each step becomes \(\text{step}_t = (\text{id}_t, I_t, s_t, A_t)\), ordered in time as \(\text{step}_1 < \cdots < \text{step}_T\), with one recording corresponding to one complete task and tasks mutually disjoint (\(T_i \cap T_j = \varnothing\)). This yields a four-level hierarchy of step → task → software → dataset. Cutting frames by state transition is the most essential choice in this pipeline: it simultaneously makes tasks compact, prevents the model from scoring repeatedly by memorizing a position on an unchanged screen, and guarantees that each step's difficulty comes from a genuine interface state rather than an annotator's segmentation taste. (Inter-annotator agreement is reported in Supplementary Material I of the original paper.)

3. First-error-terminates sequential protocol: measuring error propagation with the longest correct prefix

The evaluation rule is the sharpest part of the paper. Given frame \(I_t\), instruction \(s_t\) and interaction history \(H_t\), a model predicts click coordinates \(\hat{p}_t \sim P(p \mid I_t, s_t, H_t)\) and may also emit a set of top-\(K\) candidates \(\hat{P}_t = \{\hat{p}_{t,k}\}_{k=1}^{K}\). Ground-truth boxes are converted from normalized percentages to pixel space as \(B_t = (x_1, y_1, x_2, y_2)\), and the decision rule is deliberately plain: a prediction counts as correct if any predicted point falls inside the box (\(x_1 \le \hat{x} \le x_2\) and \(y_1 \le \hat{y} \le y_2\)). For models operating on discrete visual patches, a tolerance region \(\hat{B}_p = (\hat{x} \pm \delta, \hat{y} \pm \delta)\) is additionally allowed, and a hit is accepted if it overlaps the ground-truth box. That tolerance is a necessary engineering compromise; without it, patch-granularity models would be systematically judged wrong by grid alignment alone.

The sequential protocol itself is written as a prefix condition. Let per-step correctness be \(\delta_t = 1\) iff \(\hat{p}_t \in B_t\), take the longest correct prefix \(k = \max\{t \mid \forall i \le t, \delta_i = 1\}\), terminate evaluation of the task at the first failure, and count the task as complete only if \(k = T\) (every step correct). The authors give a formal justification: this is equivalent to maximizing the joint probability \(\prod_{t=1}^{T} P(\hat{p}_t \in B_t \mid H_t)\), so with per-step accuracy \(p\) the probability of full task completion decays roughly as \(p^{T}\). That relation shows why single-step scores overstate true reliability — on a \(T\)-step task, even 80% accuracy per step leaves only about 51% chance of being fully correct, and in a two-step task an error on the first step makes the score unreachable. Conventional practice, by contrast, averages steps independently, which charges the same loss for failing on step one and failing on the last step and thereby hides the destructiveness of early errors in clinical procedures.

The four accompanying metrics each take a different view. TCA (Task Completion Accuracy) is the fraction of tasks whose steps are all correct, the most stringent measure. S1A (Step-1 Accuracy) checks only whether the first step is grounded correctly, isolating initial perception and instruction alignment from sequential compounding. SHR (Step Hit Rate) is the ratio of steps correctly completed before the first failure to all evaluable steps, quantifying sequential progress under early stopping. WPS (Weighted Prefix Score) is newly introduced by the authors and rewards early correctness with exponential decay:

\[\text{WPS} = \sum_{i=1}^{n} \alpha^{i-1} s_i, \quad s_i = 1 \text{ if step } i \text{ is correct}, \quad \alpha = 0.8\]

It measures how long the correct prefix is and how early it sits — two correct steps at the start and two correct steps at the end do not score the same — so WPS penalizes early errors continuously, whereas TCA is binary. ⚠️ The original writes \(n\) as the number of steps in the task and \(\alpha = 0.8\) as the exponential decay factor; GUI-Actor's 1.2 and the human average of 2.167 in the tables both fall within the theoretical range for two- and three-step tasks (upper bounds 1.8 and 2.44), consistent with the formula.

4. Six-class failure taxonomy: locating errors in interface structure rather than a vague "wrong prediction"

Binary correctness cannot explain why models fail on medical interfaces, so the authors define six failure labels spanning spatial precision and semantic confusion: Edge Bias (predictions systematically drift toward the interface border, usually a broken attention prior over dense toolbar regions), Far Miss (far from the ground-truth box, i.e. no localization at all), Near Miss (very close but outside the box, exposing the precision limit of patch granularity or coordinate regression), No Prediction (the model emits no valid coordinates, common in smaller models with unstable output formatting), Small Target (failure on compact widgets), and Toolbar Confusion (clicking an adjacent or visually similar toolbar icon — the most characteristic error in medical viewers). The value of this taxonomy is diagnosability: the same 0% TCA can reflect two entirely different pathologies — a model that cannot emit coordinates at all versus one that localizes well but confuses neighboring icons — and their remedies are entirely different, the former needing output-format fixes and more training data, the latter requiring finer-grained icon semantics. The recorded distributions confirm this: small general-purpose MLLMs concentrate their errors sharply (LLaMA mostly No Prediction, DeepSeek-VL2 mostly Far Miss), while interface-specialized models (GUI-Actor, UI-TARS, Aguvis, OS-Atlas) fail in a more spread-out, difficulty-driven pattern.

A Worked Example

Figure 8 of the original paper provides a three-step DICOMscope task, "Delete the selected study from the browser menu," which shows exactly how the early-termination protocol bites. Step 1 instructs "Select a study from the 'Browser' menu" with ground-truth box \([2.7, 15.5, 11.05, 2.38]\) in percentage coordinates; GUI-Actor emits the normalized point \([0.0736, 0.1765]\) (i.e. 7.36%, 17.65%), which lands inside the box and passes, while GPT-5 emits absolute pixels \((291, 151)\) and is scored a Near Miss — the task terminates at the very first step and the deletion and confirmation steps are never evaluated. Step 2 instructs "Click on the 'Delete' Icon from the toolbar" with ground-truth box \([9.38, 5.75, 1.67, 2.98]\), only 1.67% of the image width and therefore a typical small-target toolbar icon; UI-TARS predicts \((135, 80)\) and CogAgent predicts \([0.119, 0.07]\), both scored Toolbar Confusion — they click an adjacent toolbar widget, again triggering early termination. Step 3 instructs "Confirm the deletion of the selected study" with ground-truth box \([47.21, 52.38, 2.34, 2.98]\), and only GUI-Actor reaches this point and hits, making it the single model in the figure that completes the whole task (TCA = 1).

The example also exposes an engineering detail: the four models do not share a coordinate convention — GUI-Actor and CogAgent emit 0-1 normalized coordinates, whereas UI-TARS and GPT-5 emit absolute pixels. Evaluation must interpret each output under its own convention, otherwise a formatting difference is misread as a grounding-capability difference. It also illustrates the paper's thesis most directly: GUI-Actor does not dominate every individual step, but it is the only model that does not err early, which the early-termination protocol amplifies into sole success — precisely the property workflow-aware evaluation is designed to reward.

Key Experimental Results

Main Results

The paper evaluates 16 MLLMs on MedSPOT (2 closed-source and 14 open-source), all under the strict sequential protocol (SHR/S1A in percent; WPS/TCA as reported):

Model Params SHR (%) ↑ S1A (%) ↑ WPS ↑ TCA (%) ↑
GPT-4o mini – 4.91 5.14 0.051 0.0
GPT-5 – 16.4 16.5 0.185 2.8
Llama 3.2 Vision-Instruct 11B 0.0 0.0 0.0 0.0
Qwen2-VL-Instruct 7B 1.83 1.87 0.02 0.0
DeepSeek-VL2 16B 2.7 2.8 0.03 0.0
Gemma 3 27B 3.7 1.3 0.04 0.0
Mistral-3-Instruct 8B 6.14 6.0 0.064 0.0
UGround-V1 7B 10.17 8.88 0.107 0.93
SeeClick – 4.3 9.3 0.11 1.4
Qwen2.5-VL-Instruct 7B 19.3 33.17 0.48 12.6
CogAgent 9B 37.8 27.1 0.45 15.4
Aguvis 7B 54.8 44 0.75 26.7
OS-Atlas 7B 55 45 0.8 26.6
UI-TARS 1.5-VL 7B 66 57.5 1.0 30.8
Qwen3-VL-Instruct 8B 46.6 63.0 1.1 35.0
GUI-Actor 7B 49.6 65.0 1.2 43.5

To check whether the benchmark is tractable at all, the authors also ran a small human baseline study: four annotators evaluated 45 tasks under the identical screenshots, instructions and sequential early-termination protocol.

Model TCA (%) S1A (%) SHR (%) WPS
Human average (4 annotators) 82.22 94.44 94.06 2.167
GUI-Actor (best model) 43.50 65.00 49.60 1.200

Ablation Study

(a) Sequential versus single-step evaluation. The authors compare two protocols: Sequential, where a task counts as complete only if all steps succeed in order under early termination, and Single-Step, where each step is scored independently with no enforced temporal dependency. Figure 7a shows sequential scores below single-step scores for every model, with gaps ranging from roughly 2 percentage points to roughly 35; the stronger models show the largest gaps (GUI-Actor, Qwen3-VL and UI-TARS in that tier), because their single-step ability is already decent and the remaining loss is entirely sequential compounding. This means using single-step accuracy as a proxy for task reliability systematically overestimates model capability. ⚠️ The per-model gap values in Figure 7a are typeset as overlapping small print in the original PDF; only the trend and magnitude are kept here, and exact values should be taken from the original paper.

(b) Target size thresholds. Steps are stratified by normalized bounding-box area into small (\(< 4\times10^{-4}\)), medium (\(4\times10^{-4}\)–\(3\times10^{-3}\)) and large (\(\ge 3\times10^{-3}\)) targets, and the thresholds are ablated under Conservative, Default (the paper's setting) and Relaxed configurations:

Threshold config Steps classified as small Note
Conservative 1 Threshold so strict the small tier is nearly empty; evaluation degenerates to two tiers
Default (ours) 311 The split used in the main experiments (\(< 4\times10^{-4}\) / \(4\times10^{-4}\)–\(3\times10^{-3}\) / \(\ge 3\times10^{-3}\))
Relaxed 1,421 Many mid-area elements fall into the small tier

Across all three configurations the model ranking stays stable and the small-target tier remains the hardest, with every model degrading markedly on the compact widgets common in dense DICOM viewers and annotation tools. The difficulty is therefore intrinsic to the interface rather than an artifact of the threshold choice. Qwen2-VL approaches near-zero accuracy in every size regime, reflecting spatial precision far short of what fine-grained medical GUI grounding requires. (⚠️ The medium/large sample counts under the three configurations appear only as bar labels in the original figure and are not converted here.)

(c) Failure taxonomy distribution. Recording a failure category for every incorrect step reveals sharply different pathologies across model families:

Model Params Edge Bias Far Miss Toolbar Confusion Near Miss No Prediction Small Target
Llama 3.2 Vision-Instruct 11B 49 7 4 0 134 19
Qwen2-VL-Instruct 7B 129 26 28 0 0 27
DeepSeek-VL2 16B 30 152 0 2 0 30
Gemma 3 27B 34 44 110 0 0 25
Mistral-3-Instruct 8B 109 64 13 2 0 26
UGround-V1 7B 62 66 55 4 0 25
SeeClick – 103 41 21 11 2 29
Qwen2.5-VL-Instruct 7B 20 40 78 10 10 26
CogAgent 9B 57 43 36 3 16 26
Aguvis 7B 42 84 31 4 0 23
OS-Atlas 7B 48 49 28 5 6 21
UI-TARS 1.5-VL 7B 20 55 30 0 0 22
Qwen3-VL-Instruct 8B 16 35 40 6 21 18
GUI-Actor 7B 26 26 21 1 24 17
GPT-4o mini 8B–10B 62 36 88 2 0 25
GPT-5 – 13 67 97 3 0 27

Key Findings

  • General-purpose MLLMs collapse almost completely on clinical GUIs. Llama 3.2 Vision-Instruct, Qwen2-VL-Instruct, DeepSeek-VL2 and Gemma 3 all reach 0% TCA despite parameter counts spanning 7B to 27B, with SHR below 6%. Closed-source models are equally fragile: GPT-4o mini reaches 0% TCA, and although GPT-5 lifts SHR to 16.5% and S1A to 16.5%, its TCA is still only 2.8% — it occasionally grounds the first step correctly but cannot sustain multi-step execution. Parameter scale and generic multimodal pretraining therefore do not automatically translate into reliable spatial grounding in structured clinical interfaces: medical GUIs demand pixel-level precision, discrimination among visually similar toolbar icons, and hierarchical interface understanding, which is not the same capability curve as natural-image reasoning.
  • The steep drop from S1A to TCA is universal. The performance hierarchy is consistently TCA ≪ SHR < S1A. Qwen3-VL-Instruct reaches 63.0% S1A but only 35.0% TCA; GUI-Actor attains the highest S1A (65.0%) yet completes only 43.5% of tasks; UI-TARS has the highest SHR (66%) but just 30.8% TCA. This is compounding error under early termination plus state dependency: even a moderate per-step error rate drives full task completion down through the \(p^{T}\) decay. Accurate first-step grounding does not imply reliable workflow execution, which is the paper's central message.
  • Interface-specialized architectures do markedly better, but still fall short. Aguvis and OS-Atlas both sit around 55% SHR / 26% TCA; CogAgent reaches 37.8% SHR but only 15.4% TCA, showing heavier degradation under multi-step constraints; even GUI-Actor, the strongest overall, completes only 43.5% of tasks. The most advanced models still fail more than half the benchmark, so workflow-aware grounding on high-stakes medical GUIs remains open.
  • WPS tracks TCA closely and captures "correct early." GUI-Actor takes both the highest WPS (1.2) and the highest TCA (43.5%), followed by Qwen3-VL-Instruct (1.1 / 35.0%) and UI-TARS (1.0 / 30.8%); general-purpose MLLMs sit near-zero WPS, reflecting failure in the earliest steps. WPS compensates for TCA's binary coarseness: it rewards failing late, which makes it better suited to observing partial procedural competence.
  • Failure modes are strongly structured by model family. LLaMA is dominated by No Prediction (134, the bulk of its failures), DeepSeek-VL2 by Far Miss (152), and Qwen2-VL and Mistral-3 by Edge Bias (129 / 109); on the closed-source side GPT-5 concentrates on Toolbar Confusion (97) and Far Miss (67), and GPT-4o mini likewise on Toolbar Confusion (88) plus Edge Bias (62). Interface-specialized models instead spread failures more evenly, resembling normal difficulty-driven degradation. Small Target failures persist across all 16 models (on the order of 12-18%), confirming that compact toolbar icons are an intrinsic difficulty of DICOM viewers.
  • Software difficulty varies substantially. Figure 6a ranks the ten platforms by average TCA: RadiAnt and Orthanc are hardest (the main text reports RadiAnt at 5.8% and Orthanc below 10%), ITK-SNAP is easiest (30.4% average TCA in the main text), and the rest fall mostly in the 10-25% range. ⚠️ The main text and the bar labels of Figure 6a disagree (the figure marks ITKsnap at 27.2% and RadiAnt at 5.3%, against 30.4% / 5.8% in the text); both values are recorded here as they stand rather than reconciled. Per-model breakdowns also show no model generalizing uniformly: GUI-Actor peaks at 70% on ITKsnap with 58% on MicroDicom and 50% on Orthanc, Aguvis takes the single highest score (80% on ITKsnap) but fluctuates widely on denser interfaces such as RadiAnt and GinkgoCADx, and Qwen2.5-VL is moderate on 3D Slicer (20%) and BlueLight (25%) yet collapses on DICOM-intensive tools.
  • The human baseline shows the benchmark is solvable. Four annotators reach 82.22% TCA, 94.44% S1A, 94.06% SHR and 2.167 WPS, far ahead of the best model on nearly every metric (GUI-Actor at 43.5% TCA). MedSPOT is thus clearly solvable by people, and the gap reflects model capability rather than ambiguous task definitions.

Highlights & Insights

  • "Sequence" is genuinely compressed into the score rather than being an incidental property of multi-step data. Earlier multi-step GUI benchmarks score steps independently; MedSPOT turns sequential dependency into a hard scoring constraint via the prefix condition plus early termination, and backs it with the \(\prod P(\hat{p}_t \in B_t \mid H_t)\) and \(p^{T}\) formalization. This is highly transferable: any evaluation of agents with state dependencies (software operation, long-horizon robotics, multi-turn tool use) can adopt the longest-correct-prefix plus early-stopping protocol directly, at the cost of accepting lower absolute scores.
  • WPS is a cheap but effective complementary metric. Exponential decay weighting turns "which step failed" into a continuous signal, and its single hyper-parameter (\(\alpha = 0.8\)) is interpretable as "each subsequent step is worth 20% less." Compared with TCA's 0/1, it provides finer model discrimination on a small benchmark — especially valuable when only a few hundred tasks exist.
  • The failure taxonomy is more valuable as a diagnostic than as a label set. The six categories decompose a 0% TCA into distinct pathologies: No Prediction dominance indicates output-format and scale problems, Far Miss dominance means localization fails entirely, and Toolbar Confusion dominance means fine-grained icon discrimination is lacking. LLaMA and GPT-5 are both low-scoring, but the former needs more data and stabler formatting while the latter needs stronger local icon semantics — the taxonomy points at concrete remedies, and it transfers directly to other GUI evaluations.
  • Cutting keyframes by interface state transition is a reusable rule for annotation. The simple constraint "keep only decision frames where \(F_t \neq F_{t+1}\)" solves three problems at once: tasks become compact, adjacent steps cannot be gamed by remembering a position on an unchanged screen, and each step's difficulty comes from the real interface. Any work constructing an interaction benchmark from screen recordings should reuse it.
  • The medical × GUI intersection highlights something important: medical multimodal evaluation has long stopped at static images and question answering, while the ability to operate clinical software is the actual deployment bottleneck. MedSPOT makes that gap explicit, and its step-level annotations can serve directly as SFT or RL signals for future clinical GUI agents.

Limitations & Future Work

  • Limitations the authors admit: the benchmark is currently monolingual; its scale is limited (216 task videos); only click actions are supported, excluding drag, scroll and text input; evaluation is restricted to spatial grounding (whether the predicted coordinate falls inside the ground-truth box), abstracting away richer interaction semantics; and the strict early-termination protocol, while realistic, may underestimate partial procedural competence by invalidating subsequent steps after a single error.
  • Scale and statistical power: 216 tasks and 597 keyframes is markedly smaller than ScreenSpot (1,272), ScreenSpot-Pro (1,581) or OmniACT (9,802), and the per-software conclusions rest on single-digit to low-tens task counts (Ginkgo CADx has only 12 tasks), so confidence intervals on cross-platform rankings may be wide; the human baseline covers only 45 tasks. Expanding the scale or reporting confidence intervals would make claims such as "which software is hardest" more robust.
  • Each task has only 2-3 steps, whereas real clinical procedures often run a dozen or more. Although the multi-step compounding conclusion extrapolates under the \(p^{T}\) framework, the benchmark never directly measures long-horizon tasks, so the actual curve (whether models self-correct midway, whether they recover after a wrong interface jump) remains unobserved. This is the most worthwhile direction to extend.
  • Click-only correctness judging misses "right place, wrong intent" cases: any hit inside the ground-truth box counts, yet an interface may contain several visually similar but functionally different icons, so the granularity of the box determines the tolerance boundary. The paper does not elaborate on the concrete value of \(\delta\) or its effect across models (especially patch-based ones), and this bears directly on the fairness of cross-model comparison.
  • Improvement directions: extend the action space to drag, scroll and typing; add a self-recovery branch to evaluation (for instance partial credit allowing one retry); release multilingual versions; and move failure categorization from manual post-hoc archiving toward automatic detection so it can feed large-scale training loops.
  • vs ScreenSpot / ScreenSpot-Pro: they establish cross-platform (mobile/desktop/web) and high-resolution professional-software single-step grounding evaluation at a larger scale (1,272 / 1,581), and ScreenSpot-Pro introduces step-level evaluation but no sequential protocol. The differences here are twofold: the domain shifts to clinical medical software (dense hierarchical toolbars, specialized terminology, small icons), and the relation between steps changes from parallel to state-dependent plus early-terminating. The cost is an order-of-magnitude smaller scale; the gain is the first quantification of error propagation. Consequently MedSPOT's absolute scores are not directly comparable with the ScreenSpot family — the latter averages steps independently and therefore naturally reports higher numbers.
  • vs OmniACT / AssistGUI / OSWorld-G: these also contain multi-step tasks — OSWorld-G targets real open-ended computer environments, OmniACT covers 9,802 desktop and web tasks, AssistGUI targets desktop applications. What they share is independent per-step scoring, so none can answer how many downstream steps a first-step error invalidates; and their interfaces are general-purpose software, structurally far from the dense toolbars of clinical GUIs.
  • vs medical multimodal benchmarks (GMAI-MMBench / OmniMedVQA / MedSG-Bench / VividMed): these evaluate understanding, question answering and visual grounding on static medical images (VividMed already performs medical visual grounding but still on a single image), and never touch interactive operation inside software interfaces. MedSPOT fills exactly this gap, moving from "can it read the image" to "can it operate clinical software following a procedure."
  • vs GUI-specialized models (UI-TARS / GUI-Actor / Aguvis / OS-Atlas / CogAgent): these are evaluated subjects here rather than methods. What is interesting is how they diverge — GUI-Actor, with coordinate-free grounding via a dedicated <ACTOR> token, has the highest S1A (65.0%) but only 43.5% TCA, while UI-TARS has the highest SHR (66%) yet sits in the lowest TCA tier. This suggests current GUI-specialized models are still designed primarily for single-step localization accuracy, lacking explicit multi-step state tracking and self-correction. For follow-up work, making cross-step state understanding an explicit model-side component (maintaining interface-state memory, or checking consistency of already-executed actions) is a clear direction.
  • vs Chain-of-Ground (concurrent work): it improves single-step grounding through iterative reasoning and reference feedback, which is orthogonal to this paper — one improves the model, the other improves the evaluation.

Rating

  • Novelty: ⭐⭐⭐⭐ First benchmark to combine workflow-awareness with sequential grounding on clinical software interfaces; the task formulation, early-termination protocol and six-class failure taxonomy together define an evaluation dimension that did not previously exist, though no new model is proposed.
  • Experimental Thoroughness: ⭐⭐⭐⭐ 16 MLLMs plus a human baseline, two ablation axes (sequential vs single-step, target size thresholds), a failure-taxonomy distribution and a per-software analysis; coverage is complete, but the benchmark has only 216 tasks and the human baseline only 45, limiting the statistical power of per-software claims.
  • Writing Quality: ⭐⭐⭐ Motivation and protocol definitions are clear with complete metric formulas, but the main text and Figure 6a disagree on software difficulty, some tables are set in hard-to-verify overlapping small print, and Figures 5/7 are poorly legible.
  • Value: ⭐⭐⭐⭐ Provides ready-to-use step-level evaluation and training signals for clinical GUI agents; the early-termination protocol and failure taxonomy transfer to any agent evaluation with state dependencies.