GameWorlds: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents¶
Conference: ECCV 2026
Paper: ECCV Official
Project: https://gameworld-project.github.io/
Area: Robotics / Embodied AI
Keywords: Game Agents / Multimodal LLM / Computer-Use Agents / Verifiable Evaluation / Embodied Decision-Making
TL;DR¶
GameWorld introduces the first standardized, state-verifiable benchmark for multimodal game agents across 34 browser games and 170 tasks, decoupling inference latency from execution via a paused sandbox and unifying computer-use and generalist interfaces to reveal profound capability gaps compared to human players.
Background & Motivation¶
Video games integrate high-frequency visual perception, strategic multi-stage planning, precise motor timing, and long-horizon closed-loop interaction, serving as an ideal proving ground for evaluating multimodal large language models (MLLMs) as embodied generalist agents. Unlike static visual question answering or single-turn tool invocation, gameplay forces an agent to interpret dynamic visual observations continuously, commit to irreversible physical decisions, and recover from mistakes across dozens or hundreds of sequential actions. Lightweight browser games are particularly compelling for benchmarking due to their wide variety of mechanics, lightweight rendering, and instant reconfigurability compared to heavyweight game engines.
However, existing interactive agent evaluation benchmarks remain hindered by three critical bottlenecks. First, interface fragmentation and abstraction discrepancies create severe evaluation divides across model families: some models emit raw keyboard and mouse events (Computer-Use Agents), while others operate over predefined semantic action tools (Generalist Agents), making principled cross-model comparison impossible without a unified protocol. Second, physical runtime latency confounds cognitive capability: in real-time games, an agent's multi-second inference delay fundamentally alters the incoming environment state, conflating decision quality with raw execution speed. Third, evaluation pipelines suffer from significant perceptual noise: most prior game benchmarks rely on heuristic pixel matching, OCR score extraction, or subjective VLM-as-judge scoring, introducing irreproducible variance and hallucinated grading.
To establish reliable evaluation infrastructure as a first-class design goal, the core idea of this paper is to construct GameWorld, a standardized and state-verifiable browser game benchmark that decouples inference latency via a physical clock-pausing sandbox, unifies low-level computer-use and high-level semantic actions into a shared execution protocol, and directly extracts ground-truth game states through an injected JavaScript bridge for deterministic, zero-noise outcome verification.
Method¶
Overall Architecture¶
GameWorld establishes a comprehensive closed-loop benchmarking framework comprising four foundational components: (i) Game Agent Interface Specification, which accommodates both raw low-level computer-use controls and high-level semantic tool calls; (ii) Browser-based Sandbox Runtime, which freezes the in-game physical clock during model inference and enforces deterministic seed management; (iii) Structured Game and Task Library, encompassing 34 browser games across 5 genres and 170 goal-directed tasks; and (iv) Outcome-based State-Verifiable Evaluator, which interfaces directly with serialized internal game states via an injected JavaScript bridge to yield deterministic oracle scores.
The system operates in a rigorous observation-action-verification cycle consisting of screenshot capture, readiness gating, inference pause, action normalization and execution, and internal state progress verification, as illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Environment Screenshot Observation"] --> B["Dual-Interface Action Normalization & Parsing"]
B --> C["Execution in Time-Frozen Browser Sandbox"]
C --> D["JS Bridge State Inspection & Oracle Verification"]
D --> E["Five-Level Capability Curriculum Profiling"]Key Designs¶
1. Dual-Interface Action Normalization and Deterministic Semantic Action Parsing: Unifying Heterogeneous Action Spaces
To resolve format divergence and abstraction discrepancies across model providers, GameWorld defines two distinct agent paradigms and projects them into a shared executable space:
- Computer-Use Agents (CUAs): Emit low-level keyboard and mouse primitives, such as mouse_move(x, y), left_click(x, y), press_key(key), and press_keys(key1, key2). These agents bear full responsibility for fine-grained spatial grounding and key sequence timing directly from raw visual pixels.
- Generalist Multimodal Agents: Operate within high-level semantic action spaces (e.g., move_right(), jump(), weapon_fire()). To ensure fair evaluation under the identical runtime executor, the benchmark introduces Deterministic Semantic Action Parsing, which employs a fixed, deterministic mapping from each semantic action to an exact low-level key-press event sequence, eliminating parser-side stochasticity.
To ensure consistency across models, the environment enforces a strict "One-Action-per-Step Constraint" and action atomicity: each step must yield exactly one valid interaction command, and out-of-space commands or OS-level system calls are rejected.
2. Physical Clock-Pausing Sandbox and State Readiness Gate: Decoupling Latency from Gameplay Competence
In real-time dynamic environments, model inference latency (typically ranging from 2 to 15 seconds) directly degrades the world state, causing the character to crash or fall while the model is "thinking." This conflates raw throughput speed with strategic reasoning. GameWorld decouples these factors via a Paused Evaluation Protocol: upon capturing a screenshot, the sandbox suspends the internal game loop and physics engine. Once the agent emits its action, the game execution resumes for a calibrated execution duration (typically 200–500 ms per step). This ensures that every model encounters identical environmental dynamics regardless of inference latency. Furthermore, the sandbox incorporates a Readiness Gate that monitors internal status flags upon initialization or restart, absorbing transient menu screens and loading phases so that agents always begin reasoning from an actionable gameplay state.
3. Direct State Extraction via JS Bridge and Reset-on-Fail Protocol: Eliminating Perceptual Noise
Unlike prior benchmarks that rely on error-prone OCR or subjective VLM judges, GameWorld injects a structured JavaScript bridge into the browser runtime to access the serialized gameAPI internal state directly. Across all 34 games, the benchmark exposes 233 task-relevant state fields (averaging 6.85 fields per game), covering player coordinates, health status, score counters, collected items, and checkpoint indices.
The evaluator deterministically computes two core metrics: Success Rate (SR) and normalized Progress (PG). For run \(i\) with initial score \(b_i\), target score \(\tau_i\), and maximum observed score \(q_i^{\max} = \max_t q_{i,t}\), progress is calculated as:
$\(\mathrm{progress}_i = \mathrm{clip}_{[0,1]}\!\left(\frac{q_i^{\max} - b_i}{\tau_i - b_i}\right)\)$
To prevent single accidental missteps (e.g., falling into a hazard early) from completely wiping out valid task progression, the benchmark introduces a Reset-on-Fail Protocol: upon a terminal death, the environment resets local game state, but the agent continues until its 100-action budget is exhausted, retaining the highest progress achieved across attempts.
4. Capability-Aligned Five-Level Curriculum: Comprehensive Cognitive Bottleneck Diagnosis
To systematically diagnose failure modes, GameWorld organizes all tasks into a five-level cognitive difficulty curriculum: - Level-1 (Basic Control and Timing Grounding): Atomic motor execution under minimal strategic overhead (e.g., Breakout, Core-ball, Stack). - Level-2 (System-1 Reactive Control): High-frequency reflexes and rapid evasion under continuous physical dynamics (e.g., Chrome-dino, Flappy-bird, Run-3). - Level-3 (System-2 Spatial Navigation): Geometry-aware pathfinding and hazard avoidance in structured topologies (e.g., Mario-game, Pacman, World's Hardest Game 2). - Level-4 (Symbolic Reasoning & Strategy): Rule-intensive planning with long-term consequences in discrete state spaces (e.g., 2048, Minesweeper, Wordle). - Level-5 (Open-World Coordination & Management): High-dimensional, multi-objective resource balancing and policy persistence (e.g., Minecraft Clone, Monkey Mart).
A Worked Example: Coin Collection in Mario-Game¶
Consider a task requiring Mario to navigate terrain, jump onto a platform, and strike a question block for coins while avoiding enemies:
1. Readiness Verification: The sandbox initializes the game, and the readiness gate confirms game_state.alive == true and level == "1-1".
2. Observation & Freeze: A screenshot is captured, and the browser sandbox freezes the physics engine.
3. Action Generation:
- CUA: Emits press_keys(keys=['ArrowUp', 'ArrowRight']) based on visual localization of the question block.
- Generalist: Emits jump_right(), which is mapped deterministically to press_keys(keys=['ArrowUp', 'ArrowRight'], duration=0.25).
4. Execution & State Bridge Tracking: The sandbox unfreezes for 250 ms. Mario strikes the block. The JS bridge registers state changes: game_state.score jumps from 400 to 1400, and game_state.player.x updates from 453.09 to 480.78.
5. Deterministic Evaluation: The evaluator detects that the target field coins_collected incremented, updating task progress to 1.00 without any vision or OCR heuristics.
Key Experimental Results¶
Main Results: Evaluation Across 18 Model-Interface Pairs on 5 Game Genres¶
The benchmark evaluates 13 foundation models across 18 model-interface configurations (8 CUAs and 10 Generalist agents) against both Novice and Expert human players under the standardized paused protocol with a 100-action budget.
| Model / Evaluated Agent | Interface | Arcade (SR / PG) | Platformer (SR / PG) | Puzzle (SR / PG) | Runner (SR / PG) | Simulation (SR / PG) | Overall SR (%) | Overall PG (%) | Overall Rank |
|---|---|---|---|---|---|---|---|---|---|
| Novice Player (Human) | Human | 45.7 / 55.5 | 60.0 / 65.6 | 51.4 / 63.1 | 60.0 / 72.0 | 60.0 / 62.0 | 55.3 | 64.1 | – |
| Expert Player (Human) | Human | 65.7 / 73.9 | 85.0 / 88.0 | 68.6 / 77.1 | 82.5 / 87.8 | 85.0 / 86.0 | 77.1 | 82.6 | – |
| Gemini-3-Flash-Preview | Generalist | 5.7 / 26.3 | 25.0 / 41.2 | 25.7 / 54.8 | 32.5 / 55.4 | 10.0 / 21.1 | 21.2 | 41.9 | 1 |
| GPT-5.2 | Generalist | 8.6 / 29.3 | 22.5 / 36.7 | 28.6 / 56.2 | 27.5 / 52.6 | 10.0 / 16.9 | 20.6 | 40.6 | 2 |
| Seed-1.8 | CUA | 8.6 / 31.1 | 25.0 / 40.3 | 25.7 / 52.0 | 27.5 / 50.6 | 5.0 / 11.0 | 20.0 | 39.8 | 3 |
| Claude-Sonnet-4.6 | Generalist | 5.7 / 28.3 | 22.5 / 37.0 | 25.7 / 51.5 | 30.0 / 51.9 | 15.0 / 16.6 | 20.6 | 39.3 | 4 |
| Seed-1.8 | Generalist | 11.4 / 33.5 | 22.5 / 34.6 | 22.9 / 48.7 | 27.5 / 51.2 | 10.0 / 18.8 | 20.0 | 39.0 | 5 |
| Claude-Sonnet-4.6 | CUA | 8.6 / 27.2 | 22.5 / 36.5 | 20.0 / 43.8 | 30.0 / 55.6 | 10.0 / 16.8 | 19.4 | 38.3 | 6 |
| Kimi-K2.5 | Generalist | 8.6 / 26.4 | 20.0 / 35.3 | 25.7 / 51.4 | 27.5 / 49.7 | 5.0 / 11.7 | 18.8 | 37.4 | 7 |
| Gemini-2.5-Computer-Use | CUA | 5.7 / 28.0 | 20.0 / 35.8 | 11.4 / 32.2 | 30.0 / 55.4 | 10.0 / 19.3 | 16.5 | 36.1 | 8 |
| Grok-4.1-Fast-Reasoning | Generalist | 8.6 / 23.7 | 22.5 / 37.3 | 14.3 / 46.6 | 25.0 / 49.0 | 5.0 / 10.4 | 16.5 | 36.0 | 9 |
| OpenAI-Computer-Use | CUA | 5.7 / 24.7 | 17.5 / 31.3 | 20.0 / 45.8 | 27.5 / 53.0 | 5.0 / 12.0 | 16.5 | 35.8 | 10 |
| Qwen3-VL-Plus | Generalist | 8.6 / 25.6 | 22.5 / 37.8 | 14.3 / 39.1 | 27.5 / 51.1 | 0.0 / 10.0 | 16.5 | 35.4 | 11 |
| Qwen3-VL-Plus | CUA | 5.7 / 23.5 | 20.0 / 34.6 | 14.3 / 35.6 | 27.5 / 51.0 | 5.0 / 10.7 | 15.9 | 33.6 | 12 |
| Qwen3-VL-235B-A22B | CUA | 5.7 / 23.2 | 22.5 / 35.2 | 8.6 / 29.7 | 25.0 / 51.0 | 0.0 / 1.7 | 14.1 | 31.4 | 13 |
| UI-TARS-1.5-7B | CUA | 5.7 / 31.4 | 15.0 / 24.4 | 5.7 / 29.9 | 27.5 / 52.4 | 0.0 / 3.8 | 12.4 | 31.1 | 14 |
| Qwen3-VL-235B-A22B | Generalist | 5.7 / 23.2 | 17.5 / 29.5 | 8.6 / 33.3 | 27.5 / 50.4 | 0.0 / 3.6 | 13.5 | 30.8 | 15 |
| GLM-4.6V | Generalist | 8.6 / 22.8 | 20.0 / 33.9 | 5.7 / 29.1 | 27.5 / 49.1 | 0.0 / 5.3 | 14.1 | 30.8 | 16 |
| Qwen3-VL-30B-A3B | CUA | 8.6 / 26.8 | 20.0 / 31.9 | 2.9 / 27.6 | 25.0 / 50.3 | 0.0 / 2.2 | 12.9 | 30.8 | 17 |
| Qwen3-VL-30B-A3B | Generalist | 2.9 / 20.3 | 20.0 / 36.5 | 2.9 / 26.1 | 27.5 / 51.1 | 0.0 / 3.5 | 12.4 | 30.6 | 18 |
Ablation Study & Diagnostics: Real-Time Play and Context-Memory Sensitivity¶
To explore unpaused real-time execution dynamics and memory trace scaling, the authors conducted comprehensive diagnostic ablations.
Table 1: Real-time unpaused execution (GameWorld-RT) latency and performance | Model | Interface | Step Latency (s/step) | Success Rate SR (%) | Progress PG (%) | |---|---|---|---|---| | Qwen3-VL-235B-A22B | CUA | 6.2 | 17.1 | 33.2 | | Qwen3-VL-30B-A3B | CUA | 2.4 | 15.6 | 33.0 | | Qwen3-VL-235B-A22B | Generalist | 6.4 | 16.8 | 34.0 | | Qwen3-VL-30B-A3B | Generalist | 3.4 | 15.6 | 32.9 |
Table 2: Context-memory rounds sensitivity and compute footprint (Qwen3-VL-235B-A22B) | Memory Rounds | Model Interface | Input Tokens | Step Latency (s/step) | Overall PG (%) | |---|---|---|---|---| | 0 | Generalist | 1,278 | 5.5 | 30.0 | | 0 | CUA | 1,891 | 7.2 | 30.3 | | 1 | Generalist | 2,171 | 6.8 | 30.1 | | 1 | CUA | 3,771 | 10.1 | 29.0 | | 2 | Generalist | 3,052 | 8.6 | 30.6 | | 2 | CUA | 5,627 | 12.8 | 28.7 |
Key Findings¶
- Enormous Capability Gap Against Humans: The top-performing models (Gemini-3-Flash-Preview at 21.2% SR and GPT-5.2 at 20.6% SR) remain dramatically inferior even to an untrained novice human player (55.3% SR, 64.1% PG), with a performance deficit exceeding 34% in success rate.
- Genre-Specific Capability Polarization: Models achieve relatively strong partial progress in high-frequency Runner games (PG typically 50%–55%), but collapse in complex Simulation games (SR drops to 0%–15%, with open-source models failing completely), showing that open-world multi-objective reasoning remains an unsolved frontier.
- Asymmetric Impact of Context Memory: Providing rolling trajectory memory offers marginal benefits to Generalist agents (PG rises from 30.0% to 30.6%), but degrades CUA performance (PG drops from 30.3% to 28.7%), as extended sequences of low-level coordinate traces introduce visual-spatial distraction.
- Action Validity and Instruction Following: Analyzing Invalid Action Rates (IAR) reveals that frontier proprietary models maintain 0.0% invalid actions, whereas certain open-source architectures experience significant format drifting over multi-turn interactions (e.g., GLM-4.6V exhibits 8.3% IAR, predominantly from No-Tool-Call failures).
Highlights & Insights¶
- Decoupled Physics Clock via Sandboxing: Suspending game physics during API token generation isolates decision quality from network and hardware latency, establishing an unconfounded benchmark.
- Deterministic State Inspection: Interfacing directly with internal JavaScript runtime variables eliminates the stochasticity, hallucinations, and OCR inaccuracies inherent in vision-based evaluation judges.
- Normalized Multi-Interface Protocol: Mapping both raw computer-use inputs and semantic action abstractions into a unified event space provides a standardized methodology for cross-paradigm agent comparisons.
Limitations & Future Work¶
- Author-Acknowledged Limitations: Onboarding new games currently requires manual specification of task goals, gameAPI variable instrumentation, and semantic action bindings, which caps rapid scaling. The authors propose exploring autonomous MLLM-driven environment reverse-engineering in future work.
- Self-Identified Limitations: Action durations are currently fixed to coarse discrete intervals (200–500 ms), which may constrain fine-grained physics control in games requiring sub-100 ms micro-adjustments (e.g., variable jump heights).
- Potential Extensions: Designing adaptive-duration action primitives and building automated reverse-engineering pipelines to automatically detect actionable internal memory structures.
Related Work & Insights¶
- vs OSWorld / WebArena: OSWorld and WebArena established the benchmark paradigm for execution-grounded computer-use agents in operating systems and web pages; GameWorld expands this to fast-paced, real-time gaming environments with explicit latency decoupling and internal state oracle verification.
- vs BALROG / LMGame-Bench / VideoGameBench: Prior gaming benchmarks either simplify games into text/discrete spaces (LMGame-Bench, BALROG) or rely heavily on noisy OCR and VLM heuristic evaluators (VideoGameBench); GameWorld provides the first vision-centric, state-verifiable oracle evaluation across diverse game genres.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first latency-decoupled, dual-interface, state-verifiable multimodal game benchmark.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 34 games, 170 tasks, 18 agent setups, human baselines, real-time dynamics, and memory ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous system design, transparent definitions, and self-contained failure analyses.
- Value: ⭐⭐⭐⭐⭐ Provides vital infrastructure and diagnostic foundations for developing generalist embodied decision-making agents.