Skip to content

JARVISGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / GUI Agents
Keywords: Cross-Device Coordination, GUI Agents, Type System, Dynamic Task Composition, Long-Horizon Planning

TL;DR

To overcome the single-device limitation of existing GUI agent benchmarks, this paper introduces JarvisGUI, the first large-scale dynamic benchmark spanning heterogeneous Android, Windows, and Ubuntu ecosystems; through slot-typing atomic task formalization and automatic task graph composition, JarvisGUI constructs 150 composite workflows and reveals that state-of-the-art open-source agents suffer severe performance collapses (falling as low as 0% success rate) on cross-device dependent tasks.

Background & Motivation

The rapid evolution of Multimodal Large Language Models (MLLMs) has catalyzed the development of autonomous Graphical User Interface (GUI) agents capable of perceiving complex visual interfaces and executing human-like interactions such as clicking, typing, and swiping. To rigorously measure this progress, the research community has proposed benchmarks ranging from static, single-step visual grounding suites like ScreenSpot and ScreenSpot-Pro to multi-step dynamic virtual environments including OSWorld, OmniBench, and AndroidControl. Nevertheless, almost all existing benchmarks are fundamentally grounded upon a single-device assumption, wherein every benchmark task is restricted to an isolated operating system such as Windows, Ubuntu, or Android alone.

This single-device evaluation paradigm departs sharply from realistic human digital workflows. In daily life and professional environments, users routinely coordinate tasks across smartphones, personal computers, and cloud servers—seamlessly transferring intermediate results, synchronizing files via cloud drives, and maintaining persistent task contexts across heterogeneous platforms. By neglecting cross-device dependencies and multi-ecosystem states, existing benchmarks produce an overly optimistic assessment of agent readiness for real-world deployment; meanwhile, preliminary exploratory efforts such as CRAB provide only 18 manually crafted cross-device tasks without realistic file and data transmission mechanisms, making systematic and scalable quantitative evaluation impossible.

Constructing a principled cross-device benchmark presents two fundamental hurdles: first, orchestrating multi-device environments requires standardizing heterogeneous action and observation spaces while handling variable runtime parameters and deterministic state verifications; second, automatically synthesizing multi-step workflows risks introducing meaningless operations or state-destructive conflicts (such as downstream steps inadvertently deleting or overwriting artifacts produced by upstream tasks). Core idea: develop a slot-typing-based atomic task modeling framework that treats GUI tasks as typed input-output state transformations, combine it with a three-dimensional LLM-as-a-judge filtering pipeline to guarantee task realism and evaluability, and establish the JarvisGUI benchmark on top of a four-layer virtualized container architecture, exposing critical bottlenecks in agents' inter-device state inference and long-horizon critical-path planning.

Method

Overall Architecture

The JarvisGUI system architecture is organized into four hierarchical layers: the Infrastructure Layer, Environment Control Layer, Model Interaction Layer, and Evaluation Execution Layer. The Infrastructure Layer leverages Docker containers coupled with host Linux KVM hardware acceleration modules to instantiate ephemeral, resource-isolated virtual environments across Android (via containerized ADB), Windows, and Ubuntu with checkpoint snapshots; the Environment Control Layer abstracts a Unified Observation Space (combining visual RGB screenshots with hierarchical Accessibility Trees) and a Unified Action Space (spanning cursor movement, clicks, drags, mouse scrolls, touch swipes, text typing, and meta-controls); the Model Interaction Layer addresses the single-device observation constraint of current open-source models via a two-stage Planner-Grounder architecture; finally, the Evaluation Execution Layer coordinates thread-pool scheduling and asynchronous environment pre-warming to execute dynamic evaluation and programmatic state inspection.

At the task modeling level, JarvisGUI formalizes each atomic task as a parameterized tuple \(T = (I, O, D, \Phi, P)\), where \(I\) and \(O\) denote input and output slot sequences, \(D\) is a natural language template, \(\Phi\) represents a set of deterministic evaluation functions, and \(P\) designates the target execution platform. By defining a subtype partial order \(\tau_{\text{out}} \preceq \tau_{\text{in}}\) across hierarchical data types, the system automatically stitches atomic tasks into a directed task graph \(G=(V, E)\), inserting auxiliary file-transfer tasks across platform boundaries and dynamically propagating concrete runtime values to ensure end-to-end task validity and verifiable completion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Instruction & Heterogeneous Observations<br/>Android / Windows / Ubuntu Screenshots"] --> B["Hierarchical Slot-Typing & Task Formalization<br/>T=(I, O, D, Phi, P) Dynamic Parameterization"]
    B --> C["Sampling-Based Task Graph Composition & Value Propagation<br/>Subtype Compatibility Matching + Auxiliary File Transfer Injection"]
    C --> D["Three-Dimensional LLM-as-a-Judge Filtering<br/>Authenticity · Rationality · Evaluability Scoring (Kappa=0.92)"]
    D --> E["Two-Stage Planner-Grounder Interaction Architecture<br/>Centralized Multi-Screen Planning + Single-Device Grounding"]
    E --> F["Four-Layer Sandboxed System & Deterministic State Verification<br/>KVM Virtualization + Unified Action Space + Artifact Inspection"]

Key Designs

1. Hierarchical Slot-Typing & Task Formalization: Algebraic Modeling of Inter-Task Data Flow

To prevent invalid pipeline linkages and structural data mismatches during automatic task synthesis, this work introduces a formal hierarchical type space \(\mathcal{T}\) equipped with a subtype partial order \(\preceq\). Each slot \(s \in I \cup O\) is represented as a tuple \((id, \tau)\) with \(\tau \in \mathcal{T}\) (e.g., xlsx_file \(\preceq\) ordinary_file, jpg_image \(\preceq\) image_file). When constructing directed edges in the task graph, an output slot connects to a downstream input slot if and only if subtype compatibility holds (\(\tau_{\text{out}} \preceq \tau_{\text{in}}\)). Through a parameterized mapping \((O, D, \Phi) \leftarrow \mathcal{M}(v_I)\), concrete runtime outputs generated by upstream tasks are dynamically propagated to instantiate downstream natural language placeholders (e.g., {{Slot1}}) and programmatic Python evaluation scripts, guaranteeing semantic consistency and robust execution throughout complex chains.

2. Sampling-Based Task Graph Composition & Value Propagation: Seamless Bridging Across Heterogeneous Ecosystems

To construct diverse compositional workflows at scale, JarvisGUI executes a stochastic graph composition algorithm over a human-annotated atomic task pool. Starting from an initial task node, the procedure iteratively samples open input and available output slots, querying candidate task instances that satisfy type compatibility. Crucially, whenever connected subtasks belong to different operating platforms (\(P_u \neq P_v\)), the composition engine automatically injects an auxiliary cross-platform file transfer task powered by a lightweight AList network storage server hosted within a private virtual bridge subnet. The planner is provided with structured credentials and browser interaction guidelines for AList, solving the lack of native inter-OS file bridges in automated environments.

3. Three-Dimensional LLM-as-a-Judge Filtering: Eliminating Illogical and State-Destructive Workflows

While formal type compatibility guarantees structural validity, it cannot eliminate logically absurd or un-evaluable tasks (such as deleting an image and subsequently attempting to transfer it). JarvisGUI deploys a prompt-guided LLM-as-a-judge pipeline powered by Qwen3-30B-A3B-Thinking to evaluate sampled candidates across three distinct dimensions: (1) Task Authenticity (reflecting plausible real-world user intent); (2) Task Rationality (ensuring logical coherence and avoiding superfluous actions); and (3) Evaluability (verifying that downstream actions do not destroy or overwrite observable artifacts produced by earlier steps). Only tasks scoring full marks (5/5/5) across all three axes are retained. Human expert audits over sampled tasks confirm an agreement rate of 92% and a Cohen's Kappa score of 0.92, verifying the exceptional fidelity of the filter.

4. Two-Stage Planner-Grounder Interaction Architecture: Overcoming Single-Device Visual Overload

Because state-of-the-art open-source GUI models are trained exclusively on single-device interfaces, feeding concurrent multi-device screenshots into a single end-to-end policy causes severe visual confusion. JarvisGUI decouples each interaction turn into a hierarchical Planner-Grounder paradigm. A centralized planner (instantiated with general-purpose VLMs such as Qwen3-VL-Plus) processes concurrent screenshots from Android, Windows, and Ubuntu alongside historical action plans and environment error signals to output a strictly formatted JSON decision specifying the target operating platform, an abstract step intent, and an action type (without coordinates). If the chosen action requires spatial localization (e.g., clicking or swiping), the planner delegates execution to a specialized single-platform Grounding Agent that predicts precise pixel coordinates directly on the active platform's screenshot.

Benchmark & Evaluation Protocol

JarvisGUI encompasses 118 single-platform atomic tasks (24 on Android, 56 on Ubuntu, 38 on Windows) and 150 compositional tasks (evenly partitioned into 50 Single-device with Dependencies SW, 50 Multi-device Independent MI, and 50 Multi-device Dependent MD tasks, comprising 442 subtasks and 65 auxiliary file transfer steps). Evaluation relies on two standardized metrics: - Total Task Success Rate (TSR): evaluates holistic end-to-end task completion by verifying that all terminal state evaluators \(\Phi\) evaluate to 1: $\(\text{TSR} = \frac{1}{|T|} \sum_{t \in T} \mathbf{1}_{\text{success}}(t)\)$ - Sub-task Success Rate (SSR): tracks the fine-grained completion ratio of individual subtasks within multi-step workflows: $\(\text{SSR} = \frac{1}{|T_{\text{comp}}|} \sum_{t \in T_{\text{comp}}} \frac{|s_{\text{passed}}^t|}{|s_{\text{total}}^t|}\)$

Benchmark & Experimental Results

Main Benchmark Evaluation

The benchmark comprehensively evaluates six leading open-source models: UI-Venus_Ground (7B), UI-TARS-1.5 (7B), MAI-UI (8B), GUI-OWL (32B), Qwen3-VL-Instruct (30B A3B), and HOLO2 (30B A3B), all paired with Qwen3-VL-Plus as the centralized planner. The table below presents the full results from Table 3 of the original paper:

Model Size Atomic Android (TSR) Atomic Windows (TSR) Atomic Ubuntu (TSR) Atomic Overall (TSR) Composite SW TSR / SSR Composite MI TSR / SSR Composite MD TSR / SSR Composite Overall TSR / SSR
UI-Venus_Ground 7B 33.3 18.4 51.8 37.3 4.0 / 23.2 0.0 / 24.0 0.0 / 9.7 1.3 / 16.7
UI-TARS-1.5 7B 50.0 18.4 44.6 37.3 14.0 / 28.8 4.0 / 28.0 0.0 / 8.8 6.0 / 18.8
MAI-UI 8B 37.5 7.9 39.3 28.8 8.0 / 24.0 4.0 / 27.0 0.0 / 6.5 4.0 / 16.1
GUI-OWL 32B 41.7 18.4 51.8 39.0 12.0 / 28.8 8.0 / 29.0 2.0 / 6.9 7.3 / 18.1
Qwen3-VL-Instruct 30B (A3B) 37.5 13.2 50.0 35.6 10.0 / 24.0 2.0 / 28.0 0.0 / 10.1 4.0 / 18.1
HOLO2 30B (A3B) 41.7 15.8 60.7 42.4 16.0 / 28.0 6.0 / 26.0 2.0 / 10.6 8.0 / 19.0

Note: Bold numbers indicate the best performance per column. SW: Single-device dependent; MI: Multi-device independent; MD: Multi-device dependent.

Furthermore, rigorous statistical evaluation via 95% bootstrap confidence intervals (Table 9 in the paper) highlights the severe ceiling across all models: out of 150 composite tasks, UI-Venus completes only 2 (1.3% [0.0%, 3.3%]), MAI-UI completes 6 (4.0%), Qwen3-VL completes 6 (4.0%), UI-TARS-1.5 completes 9 (6.0%), GUI-Owl completes 11 (7.3%), and HOLO2 achieves the top mark with only 12 completions (8.0% [4.0%, 12.7%]).

Alternative Planner Backbones & Generalizability

To verify whether the cross-device performance drop is an artifact of the specific planner model (Qwen3-VL-Plus), the authors conducted supplementary experiments pairing HOLO2 with alternative multimodal models (Table 8 in the paper):

Planner Backbone (HOLO2 Grounder) Atomic Android (TSR) Atomic Windows (TSR) Atomic Ubuntu (TSR) Atomic Overall (TSR) Composite SW TSR Composite MI TSR Composite MD TSR Composite Overall TSR
Qwen3-VL-Plus 41.7 15.8 60.7 42.4 16.0 6.0 2.0 8.0
Kimi K2.6 45.8 23.7 66.1 48.3 22.0 10.0 2.0 11.3
Llama 4 Maverick ~0.0 ~0.0 ~0.0 ~0.0 ~0.0 ~0.0 ~0.0 ~0.0

Note: Llama 4 Maverick exhibited limited instruction-following compliance and frequently failed to output executable actions, yielding near-zero scores across all settings.

Key Findings

  • Cross-Device Dependencies Create Fragile Critical Paths: The most catastrophic performance collapse occurs in multi-device dependent (MD) tasks, where four out of six models score 0.0% TSR and the top score is a meager 2.0%. This collapse is driven by inter-device state inference bottlenecks (models fail to track file locations across OS boundaries) and strictly sequential execution chains where any intermediate slip (such as cloud transfer login failure) causes the entire workflow to fail.
  • Multi-Device Contexts Impose Cognitive and Visual Overload: Even when subtasks are completely independent across devices (MI), success rates decline sharply compared to single-device tasks. Simultaneously processing three screen feeds degrades grounding precision, while models frequently fail in contextual platform routing (e.g., misinterpreting "C drive" as an Ubuntu path or failing to resolve anaphoric references like "back to the computer").
  • Pronounced Operating System Bias in Low-Level Robustness: Agents exhibit significantly higher execution robustness on Ubuntu (up to 60.7%) and Android (up to 50.0%) than on Windows (7.9%–18.4%). This disparity stems from pre-training data distributions rich in mobile touch interactions and Linux command/web interfaces, contrasted against scarce high-quality training trajectories for diverse Windows desktop widgets.
  • Rapid Performance Decay Across Step Horizons: Error analyses reveal that task success rates drop toward zero as composite tasks require four or more subtasks. Agents routinely exhibit premature termination or suffer from cascading failures caused by assuming unexecuted dependency steps have already succeeded.

Highlights & Insights

  • Pioneering the First Cross-Device GUI Agent Paradigm: Breaks the ubiquitous single-device assumption in GUI evaluation, systematically modeling the multi-ecosystem reality of modern digital productivity spanning smartphones, PCs, and servers.
  • Formal Algebraic Task Modeling via Hierarchical Slot Types: Employs subtype partial ordering and dynamic slot instantiation to guarantee type safety and automated argument propagation across heterogeneous tasks, enabling principled compositional scalability.
  • Robust Virtualized Multi-Platform Sandbox with AList Hub: Overcomes physical hardware testing limitations by containerizing Android, Windows, and Ubuntu under KVM acceleration, establishing an ephemeral, reproducible evaluation pipeline with cloud-based file exchange.
  • High-Agreement Three-Dimensional LLM-as-a-Judge Protocol: Enforces task authenticity, rationality, and evaluability filtering to prune out state-destructive and logically incoherent synthetic workflows, achieving a 0.92 Cohen's Kappa score with human expert judgments.

Limitations & Future Work

  • Restricted Operating System Ecosystem Coverage: Due to virtualization license constraints and closed-source proprietary software boundaries, JarvisGUI currently covers Android, Windows, and Ubuntu, omitting macOS and iOS ecosystems that represent substantial fractions of real-world user workflows.
  • Fixed Trajectory Length Truncation: To control the extreme inference costs of multi-device exploration, task trajectories are capped at 50 interaction steps, precluding evaluation of ultra-long-horizon exploration and extensive trial-and-error behaviors.
  • Simplified File Exchange Protocols: Real-world cross-device coordination relies on diverse physical protocols (e.g., physical USB mounting, AirDrop, Bluetooth pairing), whereas JarvisGUI simplifies file exchange via an AList web service with pre-configured network drive credentials.
  • Absence of Accessibility Scenarios: The benchmark focuses on mainstream productivity and power-user workflows, lacking accessibility-oriented tasks designed for elderly or disabled users (such as cross-device text-to-speech synchronization or unified accessibility setting management).
  • vs Single-Device Desktop & Mobile Benchmarks (OSWorld / AndroidControl / OmniBench): These platforms laid the groundwork for dynamic OS interaction within single domains. JarvisGUI builds upon their environment abstraction techniques while pioneering concurrent multi-platform observation, cross-OS action routing, and inter-device state dependency management.
  • vs Preliminary Multi-Device Explorations (CRAB): CRAB introduced the concept of multi-environment evaluation with 18 manual tasks but lacked realistic data and file transfer mechanisms. JarvisGUI scales this to 150 typed composite tasks with 442 subtasks and 65 file transfers, providing a fully automated, verifiable benchmark.
  • Implications for Next-Generation GUI Agents: The substantial performance gap between single-device competence and cross-device failure highlights that simply attaching off-the-shelf single-device grounders to high-level planners is inadequate; future agents require native cross-platform spatio-temporal memory and end-to-end multi-device pre-training.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first systematic cross-device GUI agent benchmark with a novel slot-typing task composition theory.
  • Experimental Rigor: ⭐⭐⭐⭐⭐ Evaluates 6 major open-source agents and 3 planner backbones across 118 atomic and 150 composite tasks with detailed bootstrap confidence intervals.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, highly informative architectural diagrams, and compelling analytical depth.
  • Impact & Value: ⭐⭐⭐⭐⭐ Directly exposes the critical capability gap between lab benchmarks and realistic multi-device deployment, providing an essential foundation for next-generation agent research.